Input Cloudflare Whitelist Scripts for Fail2ban and NGINX

[AbramS]

While migrating my Plesk server due to an OS upgrade last year, I pulled together the scripts that I wrote/modified and added some better documentation. I figured I would share these here, as there are probably some among you that can benefit from them. Hope it's helpful.
Note: I've added links to inspiration/authors wherever possible. But I couldn't quite trace the origins of some of these scripts. Happy to add a name if another author is found.

In this post I'm sharing the two scripts that I use for Cloudflare + Plesk: both are related to the Cloudflare IP-addresses. One for Fail2ban, the other for NGINX.

Cloudflare Generate Fail2ban Whitelist Script

#!/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin

# The goal of this script is to download and check the current list of Cloudflare IP-addresses and prepare a script that will add them to the Plesk Fail2ban service. This can be setup as a cronjob.
# The output script will automatically add these IP-addresses to the whitelist of the Plesk Fail2ban service. Run this script as a cronjob a couple of minutes after the first.
# Inspyr Media. Updated: 2021-07-28

# Variables
CFIPSV4=https://www.cloudflare.com/ips-v4
CFIPSV6=https://www.cloudflare.com/ips-v6
# Configure LOCALDIR to whereever you are going to store this script.
LOCALDIR=/root/cloudflare-whitelist/
LOCALIPSV4=ips-v4
LOCALIPSV6=ips-v6
WHITELIST="plesk bin ip_ban --add-trusted "
OUTPUTSCRIPT=cloudflare-whitelist-fail2ban.sh

# Set directory and clear files from previous run.
cd $LOCALDIR
rm -f $LOCALIPSV4
rm -f $LOCALIPSV6
rm -f cloudflare-ips.txt
rm -f cloudflare-cmd.txt

# Download ips-v4 file, check the file exists, is not empty and not too big.
curl -sS $CFIPSV4 >$LOCALIPSV4
sleep 3
[ ! -s $LOCALIPSV4 ] && echo "$LOCALIPSV4 does not exist. Exiting." && exit 1
if [[ -n $(find $LOCALIPSV4 -prune -size +300c) ]]
then
    echo "$LOCALIPSV4 is too big. Exiting."
    exit 1
fi
echo "$LOCALIPSV4 is ready."

# Download ips-v6 file, check the file exists, is not empty and not too big.
curl -sS $CFIPSV6 >$LOCALIPSV6
sleep 3
[ ! -s $LOCALIPSV6 ] && echo "$LOCALIPSV6 does not exist. Exiting." && exit 1
if [[ -n $(find $LOCALIPSV6 -prune -size +300c) ]]
then
    echo "$LOCALIPSV6 is too big. Exiting."
    exit 1
fi
echo "$LOCALIPSV6 is ready."

# Merge ips-v4 and ips-v6
cat $LOCALIPSV4 $LOCALIPSV6 > cloudflare-ips.txt

# Apply Plesk command as a prefix to the merged IP list.
awk -v prefix="$WHITELIST" '{print prefix $0}' cloudflare-ips.txt > cloudflare-cmd.txt

# Replace with updated version. Add shebang, path and exit to updated version and make it executable.
rm -f $OUTPUTSCRIPT
echo 'PATH=/sbin:/bin:/usr/sbin:/usr/bin' | cat - cloudflare-cmd.txt > temp && mv temp cloudflare-cmd.txt
echo '#!/bin/bash' | cat - cloudflare-cmd.txt > temp && mv temp cloudflare-cmd.txt
echo 'echo "Comitted whitelist to fail2ban."' >> cloudflare-cmd.txt
echo 'exit 0' >> cloudflare-cmd.txt
mv cloudflare-cmd.txt $OUTPUTSCRIPT
chmod 755 $OUTPUTSCRIPT
echo "Script has been generated succesfully."

# Cleanup
rm -f $LOCALIPSV4
rm -f $LOCALIPSV6
rm -f cloudflare-ips.txt
rm -f cloudflare-cmd.txt

# All done!
exit 0

Cloudflare NGINX IP Header Script

#!/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin
CFTEMP=/root/cloudflare-ips.txt
curl -sS https://www.cloudflare.com/ips-v4 >$CFTEMP
curl -sS https://www.cloudflare.com/ips-v6 >>$CFTEMP
sed -i -e 's/^/set_real_ip_from /' $CFTEMP
sed -i '1ireal_ip_header CF-Connecting-IP' $CFTEMP
sed -i '/[^;] *$/s/$/;/' $CFTEMP
mv $CFTEMP /etc/nginx/conf.d/cloudflare.conf
#check if everything is OK
nginx -t && printf "Valid\n" || printf "Error\n" | grep 'Valid' &> /dev/null
if [ $? == 0 ]; then
        echo "restaring nginx"
        service nginx restart
        echo "done"
else
        echo "something is wrong"
        mv /etc/nginx/conf.d/cloudflare.conf /etc/nginx/conf.d/cloudflare.conf-error
        echo "check cloudflare.conf-error file"
    exit 1
fi
exit 0

Disclaimer: I'm not a full-time coder and created/modified these to help solve problems that I encountered. There's probably cleaner/more efficient ways of doing this, so feel free to share any improvements.

 

[weltonw]

Thanks for sharing!

Though, if want to skip the preliminary checks, you could do:

curl https://www.cloudflare.com/ips-v{4,6} | while read -r line; do echo "Whitelisting $line"; plesk bin ip_ban --add-trusted $line; done

Arguably not as safe or readable though

 

[AbramS]

Thanks for sharing!

Though, if want to skip the preliminary checks, you could do:

curl https://www.cloudflare.com/ips-v{4,6} | while read -r line; do echo "Whitelisting $line"; plesk bin ip_ban --add-trusted $line; done

Arguably not as safe or readable though

Thanks for your input! I ended up putting those checks in place because I had occurrences where the cronjob would run over and over and over again without those checks. But that’s most certainly a good alternative.

 

[weltonw]

Well, plesk bin ip_ban --add-trusted is idempotent, so even if you did run it over and over again, you'd almost certainly run into no issue. To be fair, I'd never put what I wrote in production or recommend anyone else do that without any sanity checks

 

[Jon123]

Thanks for sharing this. I made a small change so that each IP added has a description:

#!/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin

# The goal of this script is to download and check the current list of Cloudflare IP-addresses and prepare a script that will add them to the Plesk Fail2ban service. This can be setup as a cronjob.
# The output script will automatically add these IP-addresses to the whitelist of the Plesk Fail2ban service. Run this script as a cronjob a couple of minutes after the first.
# Inspyr Media. Updated: 2021-07-28
# Jon - Updated 2023-02-27

# Variables
CFIPSV4=https://www.cloudflare.com/ips-v4
CFIPSV6=https://www.cloudflare.com/ips-v6
# Configure LOCALDIR to whereever you are going to store this script.
LOCALDIR=/root/cloudflare/
LOCALIPSV4=ips-v4
LOCALIPSV6=ips-v6
WHITELIST="plesk bin ip_ban --add-trusted "
DESCRIPTION=" -description \"CLOUDFLARE IPs - $(date +%Y-%m-%d)\""
OUTPUTSCRIPT=cloudflare-whitelist-fail2ban.sh

# Set directory and clear files from previous run.
cd $LOCALDIR
rm -f $LOCALIPSV4
rm -f $LOCALIPSV6
rm -f cloudflare-ips.txt
rm -f cloudflare-cmd.txt

# Download ips-v4 file, check the file exists, is not empty and not too big.
curl -sS $CFIPSV4 >$LOCALIPSV4
sleep 3
[ ! -s $LOCALIPSV4 ] && echo "$LOCALIPSV4 does not exist. Exiting." && exit 1
if [[ -n $(find $LOCALIPSV4 -prune -size +300c) ]]
then
    echo "$LOCALIPSV4 is too big. Exiting."
    exit 1
fi
echo "$LOCALIPSV4 is ready."

# Download ips-v6 file, check the file exists, is not empty and not too big.
curl -sS $CFIPSV6 >$LOCALIPSV6
sleep 3
[ ! -s $LOCALIPSV6 ] && echo "$LOCALIPSV6 does not exist. Exiting." && exit 1
if [[ -n $(find $LOCALIPSV6 -prune -size +300c) ]]
then
    echo "$LOCALIPSV6 is too big. Exiting."
    exit 1
fi
echo "$LOCALIPSV6 is ready."

# Merge ips-v4 and ips-v6
cat $LOCALIPSV4 $LOCALIPSV6 > cloudflare-ips.txt

# Apply Plesk command as a prefix to the merged IP list.
awk -v prefix="$WHITELIST" -v postfix="$DESCRIPTION"  '{print prefix $0 postfix}' cloudflare-ips.txt > cloudflare-cmd.txt

# Replace with updated version. Add shebang, path and exit to updated version and make it executable.
rm -f $OUTPUTSCRIPT
echo 'PATH=/sbin:/bin:/usr/sbin:/usr/bin' | cat - cloudflare-cmd.txt > temp && mv temp cloudflare-cmd.txt
echo '#!/bin/bash' | cat - cloudflare-cmd.txt > temp && mv temp cloudflare-cmd.txt
echo 'echo "Comitted whitelist to fail2ban."' >> cloudflare-cmd.txt
echo 'exit 0' >> cloudflare-cmd.txt
mv cloudflare-cmd.txt $OUTPUTSCRIPT
chmod 755 $OUTPUTSCRIPT
echo "Script has been generated succesfully."

# Cleanup
rm -f $LOCALIPSV4
rm -f $LOCALIPSV6
rm -f cloudflare-ips.txt
rm -f cloudflare-cmd.txt

# All done!
exit 0

 

[Jon123]

Since first using the script to add Cloudflare IPs to the ignore list, fail2ban was ignoring ALL ips.

The problem is here:
cat $LOCALIPSV4 $LOCALIPSV6 > cloudflare-ips.txt

If the ipv4 list from Cloudflare doesn't end with a newline, you end up with something like: "131.0.72.0/222400:cb00::/32" on that line.
In the generated script you have: "plesk bin ip_ban --add-trusted 131.0.72.0/222400:cb00::/32 -description CLOUDFLARE IPs - 2023-02-27"
And this gets added to fail2ban without error.
And apparently, fail2ban interprets that to allow all IPs.

To fix it I added a newline to the file as soon as it comes in with:
echo "" >>$LOCALIPSV4

My only concern here is if Cloudflare were to suddenly start inserting a newline at the end of the file, then we'd have a blank line. That seems to run fine, it just throws an error. It's something I'd fix if I knew how to do it off the top of my head, but I'm not worried about it.

 

[Katog Choling]

This is great. Thanks.
I must admit that I'm a bit of a newby when it comes to Linux - but I'm wondering if there's a technical reason why updating fail2ban couldn't be done with just the one script rather than having to generate and execute another to add the IPs to Fail2Ban?

 

[TomBoB]

Thanks everyone. Most appreciated. Saves us a ton of time.

 

[RenanD]

I've changed it a bit to add the scripts directly, instead of creating a new script.

#!/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin

# Variables
CFIPSV4=https://www.cloudflare.com/ips-v4
CFIPSV6=https://www.cloudflare.com/ips-v6
WHITELIST="plesk bin ip_ban --add-trusted"
DESCRIPTION="CLOUDFLARE IPs - $(date +%Y-%m-%d)"

# Function to validate IPv4 and IPv6 addresses or ranges
is_valid_ip() {
    local ip=$1
    # Match single IPv4 addresses (e.g., 192.168.1.1)
    # Match IPv4 ranges (e.g., 192.168.1.0/24)
    # Match single IPv6 addresses (e.g., 2001:0db8::1)
    # Match IPv6 ranges (e.g., 2001:0db8::/32)
    if echo "$ip" | grep -Eqo '^([0-9]{1,3}\.){3}[0-9]{1,3}(/[0-9]{1,2})?$|^[0-9a-fA-F:]+(/[0-9]{1,3})?$'; then
        return 0
    else
        return 1
    fi
}

# Function to process IPs from a given URL
process_ips() {
    local url=$1
    curl -sS "$url" | while IFS= read -r ip; do
        if is_valid_ip "$ip"; then
            $WHITELIST "$ip" -description "$DESCRIPTION"

            echo $ip
        else
            echo "Invalid IP address: $ip"
        fi
    done
}

# Process IPv4 and IPv6 addresses
echo "Processing IPv4 addresses..."
process_ips $CFIPSV4

echo "Processing IPv6 addresses..."
process_ips $CFIPSV6

echo "Whitelisted Cloudflare IPs to Fail2Ban."
exit 0

 

[AbramS]

Great to see that this topic is still alive after so many years. As I'm working through another Plesk refresh, I figured I'd have another look at my scripts and the feedback that everyone has provided over the years. This is my latest version of the script that also includes suggestions from this thread. It takes care of Fail2Ban and NGINX headers in one go. I've also added some extra validations and controls, as well as logging.

#!/bin/bash

# Constants
CF_IPV4_URL="https://www.cloudflare.com/ips-v4"
CF_IPV6_URL="https://www.cloudflare.com/ips-v6"
CF_TEMP_FILE="/tmp/cloudflare_ips.tmp"
CF_FINAL_FILE="/tmp/cloudflare_ips_final.txt"
NGINX_CF_CONF="/etc/nginx/conf.d/cloudflare.conf"
LOG_FILE="/var/log/cloudflare_ip_update.log"
TODAY=$(date '+%Y-%m-%d')
WHITELIST_REASON="Cloudflare IP whitelist - $TODAY"

# Function to log messages
log_message() {
    echo "$(date '+%Y-%m-%d %H:%M:%S') - $1" | tee -a "$LOG_FILE"
}

# Function to exit on failure
exit_on_failure() {
    log_message "ERROR: $1"
    exit 1
}

# Function to validate IP addresses (IPv4 and IPv6)
validate_ips() {
    local input_file=$1
    local valid_ips=()
    local invalid_ips=()

    # Define a combined regex for IPv4 and IPv6 validation
    local ip_regex="^(([0-9]{1,3}\.){3}[0-9]{1,3}(\/[0-9]{1,2})?)$|^([a-fA-F0-9:]+(\/[0-9]{1,3})?)$"

    while IFS= read -r ip; do
        # Skip empty lines and sanitize input
        ip=$(echo "$ip" | xargs)
        [ -z "$ip" ] && continue

        # Check against the regex
        if [[ $ip =~ $ip_regex ]]; then
            valid_ips+=("$ip")
        else
            invalid_ips+=("$ip")
            log_message "Invalid IP skipped: $ip"
        fi
    done < "$input_file"

    # Write valid IPs back to the input file
    printf "%s\n" "${valid_ips[@]}" > "$CF_FINAL_FILE"
}

# Pre-check for ngx_http_realip_module
log_message "Checking for ngx_http_realip_module..."
if ! nginx -V 2>&1 | grep -q -- '--with-http_realip_module'; then
    exit_on_failure "Nginx is not compiled with ngx_http_realip_module. Please recompile or install a compatible version."
fi

# Fetch Cloudflare IPs
log_message "Fetching Cloudflare IP lists..."
curl -s "$CF_IPV4_URL" -o /tmp/cloudflare_ips_v4.tmp &
curl -s "$CF_IPV6_URL" -o /tmp/cloudflare_ips_v6.tmp &
wait

# Validate downloads
[ -s /tmp/cloudflare_ips_v4.tmp ] || exit_on_failure "Failed to download IPv4 list."
[ -s /tmp/cloudflare_ips_v6.tmp ] || exit_on_failure "Failed to download IPv6 list."

# Combine lists and validate IPs
log_message "Validating and combining IP lists..."
cat /tmp/cloudflare_ips_v4.tmp <(echo) /tmp/cloudflare_ips_v6.tmp > "$CF_TEMP_FILE"
validate_ips "$CF_TEMP_FILE"

# Ensure valid IPs were written to final file
if [ ! -s "$CF_FINAL_FILE" ]; then
    exit_on_failure "No valid IPs found after validation."
fi

# Update Fail2Ban whitelist one IP at a time
log_message "Updating Fail2Ban whitelist individually for each IP..."
while IFS= read -r ip; do
    log_message "Adding IP $ip with reason: $WHITELIST_REASON"
    /usr/sbin/plesk bin ip_ban --add-trusted "$ip" -description "$WHITELIST_REASON" || log_message "Failed to add $ip to whitelist."
done < "$CF_FINAL_FILE"

# Update Nginx configuration for real IPs
log_message "Updating Nginx configuration for real IPs..."
{
    echo "# Cloudflare IPs"
    while IFS= read -r ip; do
        echo "set_real_ip_from $ip;"
    done < "$CF_FINAL_FILE"
    echo "real_ip_header CF-Connecting-IP;"
    echo "real_ip_recursive on;"
} > "$NGINX_CF_CONF" || exit_on_failure "Failed to write Nginx configuration."

# Test Nginx configuration
log_message "Testing Nginx configuration..."
if ! nginx -t; then
    log_message "Nginx configuration test failed. Removing $NGINX_CF_CONF."
    rm -f "$NGINX_CF_CONF"
    
    log_message "Retesting Nginx configuration after rollback..."
    if ! nginx -t; then
        exit_on_failure "Nginx configuration is still invalid after rollback. Manual intervention required."
    fi
fi

# Reload Nginx
log_message "Reloading Nginx to apply changes..."
systemctl reload nginx || exit_on_failure "Failed to reload Nginx."

# Cleanup
rm -f /tmp/cloudflare_ips_v4.tmp /tmp/cloudflare_ips_v6.tmp "$CF_TEMP_FILE"
log_message "Temporary files cleaned up."

log_message "Cloudflare IP update completed successfully."

 

[AbramS]

Was unable to edit my post anymore, but obviously need to define the PATH to make this work in crontab.

E.g. add this to line 2 of the code:

PATH="/usr/sbin:/usr/bin:/sbin:/bin"

 

[WebHostingAce]

Thank you for sharing!

This is the script I have been using for this purpose:

https://raw.githubusercontent.com/plesk/kb-scripts/master/cf-nginx-ip-passthrough/cf.sh

I have been using it based on the instructions from:

https://support.plesk.com/hc/en-us/articles/12377061126807-Incorrect-IP-addresses-are-logged-in-the-access-logs-of-Plesk-domains-behind-a-Cloudflare-CDN-proxy-or-Google-Cloud-Load-Balancing

Also, I have one question regarding your script:

After running the script, we should see the Real IP. So why do we need to add the Cloudflare IPs to Fail2Ban?