Input Cloudflare Whitelist Scripts for Fail2ban and NGINX

[AbramS]

While migrating my Plesk server due to an OS upgrade last year, I pulled together the scripts that I wrote/modified and added some better documentation. I figured I would share these here, as there are probably some among you that can benefit from them. Hope it's helpful.
Note: I've added links to inspiration/authors wherever possible. But I couldn't quite trace the origins of some of these scripts. Happy to add a name if another author is found.

In this post I'm sharing the two scripts that I use for Cloudflare + Plesk: both are related to the Cloudflare IP-addresses. One for Fail2ban, the other for NGINX.

Cloudflare Generate Fail2ban Whitelist Script

#!/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin

# The goal of this script is to download and check the current list of Cloudflare IP-addresses and prepare a script that will add them to the Plesk Fail2ban service. This can be setup as a cronjob.
# The output script will automatically add these IP-addresses to the whitelist of the Plesk Fail2ban service. Run this script as a cronjob a couple of minutes after the first.
# Inspyr Media. Updated: 2021-07-28

# Variables
CFIPSV4=https://www.cloudflare.com/ips-v4
CFIPSV6=https://www.cloudflare.com/ips-v6
# Configure LOCALDIR to whereever you are going to store this script.
LOCALDIR=/root/cloudflare-whitelist/
LOCALIPSV4=ips-v4
LOCALIPSV6=ips-v6
WHITELIST="plesk bin ip_ban --add-trusted "
OUTPUTSCRIPT=cloudflare-whitelist-fail2ban.sh

# Set directory and clear files from previous run.
cd $LOCALDIR
rm -f $LOCALIPSV4
rm -f $LOCALIPSV6
rm -f cloudflare-ips.txt
rm -f cloudflare-cmd.txt

# Download ips-v4 file, check the file exists, is not empty and not too big.
curl -sS $CFIPSV4 >$LOCALIPSV4
sleep 3
[ ! -s $LOCALIPSV4 ] && echo "$LOCALIPSV4 does not exist. Exiting." && exit 1
if [[ -n $(find $LOCALIPSV4 -prune -size +300c) ]]
then
    echo "$LOCALIPSV4 is too big. Exiting."
    exit 1
fi
echo "$LOCALIPSV4 is ready."

# Download ips-v6 file, check the file exists, is not empty and not too big.
curl -sS $CFIPSV6 >$LOCALIPSV6
sleep 3
[ ! -s $LOCALIPSV6 ] && echo "$LOCALIPSV6 does not exist. Exiting." && exit 1
if [[ -n $(find $LOCALIPSV6 -prune -size +300c) ]]
then
    echo "$LOCALIPSV6 is too big. Exiting."
    exit 1
fi
echo "$LOCALIPSV6 is ready."

# Merge ips-v4 and ips-v6
cat $LOCALIPSV4 $LOCALIPSV6 > cloudflare-ips.txt

# Apply Plesk command as a prefix to the merged IP list.
awk -v prefix="$WHITELIST" '{print prefix $0}' cloudflare-ips.txt > cloudflare-cmd.txt

# Replace with updated version. Add shebang, path and exit to updated version and make it executable.
rm -f $OUTPUTSCRIPT
echo 'PATH=/sbin:/bin:/usr/sbin:/usr/bin' | cat - cloudflare-cmd.txt > temp && mv temp cloudflare-cmd.txt
echo '#!/bin/bash' | cat - cloudflare-cmd.txt > temp && mv temp cloudflare-cmd.txt
echo 'echo "Comitted whitelist to fail2ban."' >> cloudflare-cmd.txt
echo 'exit 0' >> cloudflare-cmd.txt
mv cloudflare-cmd.txt $OUTPUTSCRIPT
chmod 755 $OUTPUTSCRIPT
echo "Script has been generated succesfully."

# Cleanup
rm -f $LOCALIPSV4
rm -f $LOCALIPSV6
rm -f cloudflare-ips.txt
rm -f cloudflare-cmd.txt

# All done!
exit 0

Cloudflare NGINX IP Header Script

#!/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin
CFTEMP=/root/cloudflare-ips.txt
curl -sS https://www.cloudflare.com/ips-v4 >$CFTEMP
curl -sS https://www.cloudflare.com/ips-v6 >>$CFTEMP
sed -i -e 's/^/set_real_ip_from /' $CFTEMP
sed -i '1ireal_ip_header CF-Connecting-IP' $CFTEMP
sed -i '/[^;] *$/s/$/;/' $CFTEMP
mv $CFTEMP /etc/nginx/conf.d/cloudflare.conf
#check if everything is OK
nginx -t && printf "Valid\n" || printf "Error\n" | grep 'Valid' &> /dev/null
if [ $? == 0 ]; then
        echo "restaring nginx"
        service nginx restart
        echo "done"
else
        echo "something is wrong"
        mv /etc/nginx/conf.d/cloudflare.conf /etc/nginx/conf.d/cloudflare.conf-error
        echo "check cloudflare.conf-error file"
    exit 1
fi
exit 0

Disclaimer: I'm not a full-time coder and created/modified these to help solve problems that I encountered. There's probably cleaner/more efficient ways of doing this, so feel free to share any improvements.

 

[weltonw]

Thanks for sharing!

Though, if want to skip the preliminary checks, you could do:

curl https://www.cloudflare.com/ips-v{4,6} | while read -r line; do echo "Whitelisting $line"; plesk bin ip_ban --add-trusted $line; done

Arguably not as safe or readable though

 

[AbramS]

Thanks for sharing!

Though, if want to skip the preliminary checks, you could do:

curl https://www.cloudflare.com/ips-v{4,6} | while read -r line; do echo "Whitelisting $line"; plesk bin ip_ban --add-trusted $line; done

Arguably not as safe or readable though

Thanks for your input! I ended up putting those checks in place because I had occurrences where the cronjob would run over and over and over again without those checks. But that’s most certainly a good alternative.

 

[weltonw]

Well, plesk bin ip_ban --add-trusted is idempotent, so even if you did run it over and over again, you'd almost certainly run into no issue. To be fair, I'd never put what I wrote in production or recommend anyone else do that without any sanity checks

 

[Jon123]

Thanks for sharing this. I made a small change so that each IP added has a description:

#!/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin

# The goal of this script is to download and check the current list of Cloudflare IP-addresses and prepare a script that will add them to the Plesk Fail2ban service. This can be setup as a cronjob.
# The output script will automatically add these IP-addresses to the whitelist of the Plesk Fail2ban service. Run this script as a cronjob a couple of minutes after the first.
# Inspyr Media. Updated: 2021-07-28
# Jon - Updated 2023-02-27

# Variables
CFIPSV4=https://www.cloudflare.com/ips-v4
CFIPSV6=https://www.cloudflare.com/ips-v6
# Configure LOCALDIR to whereever you are going to store this script.
LOCALDIR=/root/cloudflare/
LOCALIPSV4=ips-v4
LOCALIPSV6=ips-v6
WHITELIST="plesk bin ip_ban --add-trusted "
DESCRIPTION=" -description \"CLOUDFLARE IPs - $(date +%Y-%m-%d)\""
OUTPUTSCRIPT=cloudflare-whitelist-fail2ban.sh

# Set directory and clear files from previous run.
cd $LOCALDIR
rm -f $LOCALIPSV4
rm -f $LOCALIPSV6
rm -f cloudflare-ips.txt
rm -f cloudflare-cmd.txt

# Download ips-v4 file, check the file exists, is not empty and not too big.
curl -sS $CFIPSV4 >$LOCALIPSV4
sleep 3
[ ! -s $LOCALIPSV4 ] && echo "$LOCALIPSV4 does not exist. Exiting." && exit 1
if [[ -n $(find $LOCALIPSV4 -prune -size +300c) ]]
then
    echo "$LOCALIPSV4 is too big. Exiting."
    exit 1
fi
echo "$LOCALIPSV4 is ready."

# Download ips-v6 file, check the file exists, is not empty and not too big.
curl -sS $CFIPSV6 >$LOCALIPSV6
sleep 3
[ ! -s $LOCALIPSV6 ] && echo "$LOCALIPSV6 does not exist. Exiting." && exit 1
if [[ -n $(find $LOCALIPSV6 -prune -size +300c) ]]
then
    echo "$LOCALIPSV6 is too big. Exiting."
    exit 1
fi
echo "$LOCALIPSV6 is ready."

# Merge ips-v4 and ips-v6
cat $LOCALIPSV4 $LOCALIPSV6 > cloudflare-ips.txt

# Apply Plesk command as a prefix to the merged IP list.
awk -v prefix="$WHITELIST" -v postfix="$DESCRIPTION"  '{print prefix $0 postfix}' cloudflare-ips.txt > cloudflare-cmd.txt

# Replace with updated version. Add shebang, path and exit to updated version and make it executable.
rm -f $OUTPUTSCRIPT
echo 'PATH=/sbin:/bin:/usr/sbin:/usr/bin' | cat - cloudflare-cmd.txt > temp && mv temp cloudflare-cmd.txt
echo '#!/bin/bash' | cat - cloudflare-cmd.txt > temp && mv temp cloudflare-cmd.txt
echo 'echo "Comitted whitelist to fail2ban."' >> cloudflare-cmd.txt
echo 'exit 0' >> cloudflare-cmd.txt
mv cloudflare-cmd.txt $OUTPUTSCRIPT
chmod 755 $OUTPUTSCRIPT
echo "Script has been generated succesfully."

# Cleanup
rm -f $LOCALIPSV4
rm -f $LOCALIPSV6
rm -f cloudflare-ips.txt
rm -f cloudflare-cmd.txt

# All done!
exit 0

 

[Jon123]

Since first using the script to add Cloudflare IPs to the ignore list, fail2ban was ignoring ALL ips.

The problem is here:
cat $LOCALIPSV4 $LOCALIPSV6 > cloudflare-ips.txt

If the ipv4 list from Cloudflare doesn't end with a newline, you end up with something like: "131.0.72.0/222400:cb00::/32" on that line.
In the generated script you have: "plesk bin ip_ban --add-trusted 131.0.72.0/222400:cb00::/32 -description CLOUDFLARE IPs - 2023-02-27"
And this gets added to fail2ban without error.
And apparently, fail2ban interprets that to allow all IPs.

To fix it I added a newline to the file as soon as it comes in with:
echo "" >>$LOCALIPSV4

My only concern here is if Cloudflare were to suddenly start inserting a newline at the end of the file, then we'd have a blank line. That seems to run fine, it just throws an error. It's something I'd fix if I knew how to do it off the top of my head, but I'm not worried about it.

 

[Katog Choling]

This is great. Thanks.
I must admit that I'm a bit of a newby when it comes to Linux - but I'm wondering if there's a technical reason why updating fail2ban couldn't be done with just the one script rather than having to generate and execute another to add the IPs to Fail2Ban?

 

[TomBoB]

Thanks everyone. Most appreciated. Saves us a ton of time.

 

[RenanD]

I've changed it a bit to add the scripts directly, instead of creating a new script.

#!/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin

# Variables
CFIPSV4=https://www.cloudflare.com/ips-v4
CFIPSV6=https://www.cloudflare.com/ips-v6
WHITELIST="plesk bin ip_ban --add-trusted"
DESCRIPTION="CLOUDFLARE IPs - $(date +%Y-%m-%d)"

# Function to validate IPv4 and IPv6 addresses or ranges
is_valid_ip() {
    local ip=$1
    # Match single IPv4 addresses (e.g., 192.168.1.1)
    # Match IPv4 ranges (e.g., 192.168.1.0/24)
    # Match single IPv6 addresses (e.g., 2001:0db8::1)
    # Match IPv6 ranges (e.g., 2001:0db8::/32)
    if echo "$ip" | grep -Eqo '^([0-9]{1,3}\.){3}[0-9]{1,3}(/[0-9]{1,2})?$|^[0-9a-fA-F:]+(/[0-9]{1,3})?$'; then
        return 0
    else
        return 1
    fi
}

# Function to process IPs from a given URL
process_ips() {
    local url=$1
    curl -sS "$url" | while IFS= read -r ip; do
        if is_valid_ip "$ip"; then
            $WHITELIST "$ip" -description "$DESCRIPTION"

            echo $ip
        else
            echo "Invalid IP address: $ip"
        fi
    done
}

# Process IPv4 and IPv6 addresses
echo "Processing IPv4 addresses..."
process_ips $CFIPSV4

echo "Processing IPv6 addresses..."
process_ips $CFIPSV6

echo "Whitelisted Cloudflare IPs to Fail2Ban."
exit 0