Question Fail2Ban capture and ban IP, connection still being made

[Alban Staehli]

Server operating system version

CentOS 7.9

Plesk version and microupdate number

18.0.61

Hi,

Running Plesk Obsidian Version 18.0.61 on Centos 7.9.2009 (for the last week before moving to Alma) in a LXC in Proxmox.
Proxmox firewall enabled for the LXC, while iptable rules within the LXC works. From inside the Plesk LXC, if I add an iptable rule to block an IP , it actually does block the access.

As part of the apache badbot jail, I'm attempting to block Amazonbot as it does not respect robots.txt directives. The site is proxied via Cloudflare.
Fail2ban regex updated, reloaded, all good. I can now see its IP being listed as banned under Plesk UI.
Nevetherless, I also blocked via a nginx directive on the vhost itself, and still catches the connections from the meant to be blocked IP:

=> 52.70.240.171 being one of the IP

It's banned in Plesk:

I can see the ip being part of the f2b-BadBits chain in iptables
Chain f2b-BadBots (1 references)
target prot opt source destination
REJECT all -- 52.70.240.171 0.0.0.0/0 reject-with icmp-port-unreachable

I don't understand.
I verified as well that it catches custom bot and block access properly - no issue.

Any idea where shall I look?

Thanks.

 

[AYamshanov]

Hi!

Where the first screenshot was taken? I can't recognize this part of the interface.

As I understand, if the site is proxied via Cloudflare, the bad IPs do not connect to your server directly, it does Cloudflare. In this case, a firewall will not block TCP/IP packages on a network level because Cloudflare connects to the server (and uses Cloudflare IP addresses for that; not the bad one IP-address).

To let the server know what internet address is requested for a page, usually use the "X-Forwarded-For" header (Cloudflare HTTP request headers · Cloudflare Fundamentals docs). This header is a part of the next level (Application) and if you want to block addresses based on these data, you need to different way to do that.

Nevetherless, I also blocked via a nginx directive on the vhost itself, and still catches the connections from the meant to be blocked IP:

I think makes sense to review the rules one more time because if everything is configured correctly, it should help.

 

[Alban Staehli]

Thanks @AYamshanov, makes perfect sense.
Logs from Apache/Nginx shows X-Forwarded-For IP as expected, hence fail2ban bans accordingly, while Iptables will see Cloudflare IP and hence let the traffic flow through.

I will then rely on rules at nginx level for cloudflare exposed sites for now.

1st screenshot is just part of the log viewer under the domain dashboard in Plesk, not all columns are visible.

 

[Bitpalast]

View attachment 26436
=> 52.70.240.171 being one of the IP

It's banned in Plesk:
View attachment 26437
I can see the ip being part of the f2b-BadBits chain in iptables
Chain f2b-BadBots (1 references)
target prot opt source destination
REJECT all -- 52.70.240.171 0.0.0.0/0 reject-with icmp-port-unreachable

If a connection was made before the ban was installed, in some cases unwanted traffic can continue for a while. If there is a lot of traffic from one source, it can also take a minute or two until the log reflects the true current situation, because previous processes have not finished yet and have not been logged yet. So even if there is nothing new coming into the server, it can take a short while until you recognize that the ban is effective in the log files of the web server.

 

[Alban Staehli]

I actually found something interesting: Using Fail2Ban with Cloudflare on a free account
Shall be part of a Plesk extension, to sync up IP blocking list from fail2ban to cloudflare.

 

[JohnnyU]

Hello,

I'm also in this situation, having my DNS moved from namecheap to Cloudflare due to constant problems with excessive traffic and load

So I'm not 100% sure fail2ban is working because I still see with htop the cpu cores loaded to 100% This is what I get when running:
fail2ban-regex --verbose /var/www/vhosts/SITE.com/logs/access_ssl_log /etc/fail2ban/filter.d/apache-badbots.conf

    Results

    =======


    Failregex: 364 total

    |-  #) [# of hits] regular expression

    |   1) [364] ^<HOST> -[^"]*"(?:GET|POST|HEAD) \/.* HTTP\/\d(?:\.\d+)" \d+ \d+ "[^"]*" "[^"]*(?:GPTBot|AmazonBot|Bytespider|Bytedance|fidget-spinner-bot|EmailCollector|WebEMailExtrac|ClaudeBot|ClaudeBot/1\.0|TrackBack/1\.02|ImagesiftBot|PetalBot|Barkrowler|SeekportBot|serpstatbot|GeedoProductSearch|sogou music spider|seocompany|LieBaoFast|SEOkicks|Uptimebot|Cliqzbot|ssearch_bot|domaincrawler|AhrefsBot|spot|DigExt|Sogou|MegaIndex\.ru|majestic12|80legs|SISTRIX|HTTrack|Semrush|MJ12|Ezooms|CCBot|TalkTalk|Ahrefs|BLEXBot|Atomic_Email_Hunter/4\.0|atSpider/1\.0|autoemailspider|bwh3_user_agent|China Local Browse 2\.6|ContactBot/0\.2|ContentSmartz|DataCha0s/2\.0|DBrowse 1\.4b|DBrowse 1\.4d|Demo Bot DOT 16b|Demo Bot Z 16b|DSurf15a 01|DSurf15a 71|DSurf15a 81|DSurf15a VA|EBrowse 1\.4b|Educate Search VxB|EmailSiphon|EmailSpider|EmailWolf 1\.00|ESurf15a 15|ExtractorPro|Franklin Locator 1\.8|FSurf15a 01|Full Web Bot 0416B|Full Web Bot 0516B|Full Web Bot 2816B|Guestbook Auto Submitter|Industry Program 1\.0\.x|ISC Systems iRc Search 2\.1|IUPUI Research Bot v 1\.9a|LARBIN-EXPERIMENTAL \(efp@gmx\.net\)|LetsCrawl\.com/1\.0 \+http\://letscrawl\.com/|Lincoln State Web Browser|LMQueueBot/0\.2|LWP\:\:Simple/5\.803|Mac Finder 1\.0\.xx|MFC Foundation Class Library 4\.0|Microsoft URL Control - 6\.00\.8xxx|Missauga Locate 1\.0\.0|Missigua Locator 1\.9|Missouri College Browse|Mizzu Labs 2\.2|Mo College 1\.9|MVAClient|Mozilla/2\.0 \(compatible; NEWT ActiveX; Win32\)|Mozilla/3\.0 \(compatible; Indy Library\)|Mozilla/3\.0 \(compatible; scan4mail \(advanced version\) http\://www\.peterspages\.net/?scan4mail\)|Mozilla/4\.0 \(compatible; Advanced Email Extractor v2\.xx\)|Mozilla/4\.0 \(compatible; Iplexx Spider/1\.0 http\://www\.iplexx\.at\)|Mozilla/4\.0 \(compatible; MSIE 5\.0; Windows NT; DigExt; DTS Agent|Mozilla/4\.0 efp@gmx\.net|Mozilla/5\.0 \(Version\: xxxx Type\:xx\)|NameOfAgent \(CMS Spider\)|NASA Search 1\.0|Nsauditor/1\.x|PBrowse 1\.4b|PEval 1\.4b|Poirot|Port Huron Labs|Production Bot 0116B|Production Bot 2016B|Production Bot DOT 3016B|Program Shareware 1\.0\.2|PSurf15a 11|PSurf15a 51|PSurf15a VA|psycheclone|RSurf15a 41|RSurf15a 51|RSurf15a 81|searchbot admin@google\.com|ShablastBot 1\.0|snap\.com beta crawler v0|Snapbot/1\.0|Snapbot/1\.0 \(Snap Shots&#44; \+http\://www\.snap\.com\)|sogou develop spider|Sogou Orion spider/3\.0\(\+http\://www\.sogou\.com/docs/help/webmasters\.htm#07\)|sogou spider|Sogou web spider/3\.0\(\+http\://www\.sogou\.com/docs/help/webmasters\.htm#07\)|sohu agent|SSurf15a 11 |TSurf15a 11|Under the Rainbow 2\.2|User-Agent\: Mozilla/4\.0 \(compatible; MSIE 6\.0; Windows NT 5\.1\)|VadixBot|WebVulnCrawl\.unknown/1\.0 libwww-perl/5\.803|Wells Search II|WEP Search 00|thesis-research-bot)[^"]*"$

    |      47.128.59.106  Mon Jun 24 04:11:17 2024
    |      47.128.59.106  Mon Jun 24 04:11:19 2024
    |      47.128.19.152  Mon Jun 24 04:11:21 2024
    |      47.128.98.115  Mon Jun 24 04:11:25 2024
    |      47.128.123.46  Mon Jun 24 04:11:28 2024
    |      47.128.61.6  Mon Jun 24 04:11:40 2024
    |      47.128.49.180  Mon Jun 24 04:12:32 2024
    |      47.128.123.97  Mon Jun 24 04:13:00 2024
    |      47.128.46.11  Mon Jun 24 04:13:30 2024
    |      47.128.46.11  Mon Jun 24 04:13:35 2024
    |      47.128.46.11  Mon Jun 24 04:13:35 2024
    |      47.128.46.11  Mon Jun 24 04:13:35 2024
    |      47.128.23.116  Mon Jun 24 04:13:49 2024
    |      47.128.23.116  Mon Jun 24 04:13:53 2024
    |      47.128.23.116  Mon Jun 24 04:13:53 2024
    |      47.128.23.116  Mon Jun 24 04:13:53 2024
    |      47.128.23.116  Mon Jun 24 04:13:53 2024
    |      47.128.23.116  Mon Jun 24 04:13:53 2024
    |      47.128.23.116  Mon Jun 24 04:13:53 2024
    |      47.128.23.116  Mon Jun 24 04:13:53 2024
    |      47.128.25.21  Mon Jun 24 04:14:16 2024
    |      47.128.58.223  Mon Jun 24 04:14:34 2024
    |      47.128.58.223  Mon Jun 24 04:14:36 2024
    |      47.128.126.31  Mon Jun 24 04:14:57 2024
    |      47.128.59.65  Mon Jun 24 04:15:19 2024
    |      47.128.59.65  Mon Jun 24 04:15:20 2024
    |      47.128.59.69  Mon Jun 24 04:15:43 2024
    |      52.230.152.100  Mon Jun 24 04:15:59 2024
    |      47.128.52.54  Mon Jun 24 04:16:03 2024
    |      47.128.59.85  Mon Jun 24 04:16:28 2024
    |      52.230.152.205  Mon Jun 24 04:16:59 2024
    |      47.128.63.99  Mon Jun 24 04:17:10 2024
    |      47.128.56.157  Mon Jun 24 04:17:33 2024
    |      47.128.36.234  Mon Jun 24 04:18:16 2024
    |      47.128.36.162  Mon Jun 24 04:18:23 2024
    |      47.128.29.205  Mon Jun 24 04:18:26 2024
    |      47.128.17.97  Mon Jun 24 04:18:30 2024
    |      47.128.122.54  Mon Jun 24 04:18:33 2024
    |      47.128.38.117  Mon Jun 24 04:28:55 2024
    |      47.128.62.210  Mon Jun 24 04:28:58 2024
    |      47.128.18.104  Mon Jun 24 04:29:01 2024
    |      47.128.121.174  Mon Jun 24 04:29:03 2024
    |      47.128.26.98  Mon Jun 24 04:29:06 2024
    |      47.128.54.160  Mon Jun 24 04:29:09 2024
    |      47.128.23.160  Mon Jun 24 04:29:13 2024
    |      47.128.58.91  Mon Jun 24 04:29:13 2024
    |      47.128.123.222  Mon Jun 24 04:29:16 2024
    |      47.128.49.34  Mon Jun 24 04:29:19 2024
    |      47.128.59.212  Mon Jun 24 04:32:00 2024
    |      47.128.59.225  Mon Jun 24 04:32:04 2024
    |      47.128.59.193  Mon Jun 24 04:32:09 2024
    |      47.128.55.157  Mon Jun 24 04:32:10 2024
    |      47.128.99.117  Mon Jun 24 04:32:14 2024
    |      47.128.99.122  Mon Jun 24 04:32:17 2024
    |      47.128.98.88  Mon Jun 24 04:32:19 2024
    |      47.128.110.160  Mon Jun 24 04:32:21 2024
    |      47.128.99.10  Mon Jun 24 04:32:24 2024
    |      47.128.34.176  Mon Jun 24 04:43:29 2024

    ..........................................................................................
  

    Ignoreregex: 0 total


    Date template hits:

    |- [# of hits] date format

    |  [161865] ^[^\[]*\[(Day(?P<_sep>[-/])MON(?P=_sep)ExYear[ :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone offset)?)
    |  [0] ^[^\[]*\[(ExYear(?P<_sep>[-/.])Month(?P=_sep)Day(?:T|  ?)24hour:Minute:Second(?:[.,]Microseconds)?(?:\s*Zone offset)?)
    |  [0] ^[^\[]*\[((?:DAY )?MON Day %k:Minute:Second(?:\.Microseconds)?(?: ExYear)?)
    |  [0] ^[^\[]*\[((?:DAY )?MON Day ExYear %k:Minute:Second(?:\.Microseconds)?)
    |  [0] ^[^\[]*\[(Day(?P<_sep>[-/])Month(?P=_sep)(?:ExYear|ExYear2) %k:Minute:Second)
    |  [0] ^[^\[]*\[(Month/Day/ExYear:24hour:Minute:Second)
    |  [0] ^[^\[]*\[(Month-Day-ExYear %k:Minute:Second(?:\.Microseconds)?)
    |  [0] ^[^\[]*\[(Epoch)
    |  [0] ^[^\[]*\[(24hour:Minute:Second)
    |  [0] ^[^\[]*\[(^<Month/Day/ExYear2@24hour:Minute:Second>)
    |  [0] ^[^\[]*\[(ExYear2ExMonthExDay  ?24hour:Minute:Second)
    |  [0] ^[^\[]*\[(MON Day, ExYear 12hour:Minute:Second AMPM)
    |  [0] ^[^\[]*\[(^MON-Day-ExYear2 %k:Minute:Second)
    |  [0] ^[^\[]*\[(ExYearExMonthExDay(?:T|  ?)Ex24hourExMinuteExSecond(?:[.,]Microseconds)?(?:\s*Zone offset)?)
    |  [0] ^[^\[]*\[((?:Zone name )?(?:DAY )?MON Day %k:Minute:Second(?:\.Microseconds)?(?: ExYear)?)
    |  [0] ^[^\[]*\[((?:Zone offset )?(?:DAY )?MON Day %k:Minute:Second(?:\.Microseconds)?(?: ExYear)?)
    |  [0] ^[^\[]*\[(TAI64N)
    |  [0] {^LN-BEG}ExYear(?P<_sep>[-/.])Month(?P=_sep)Day(?:T|  ?)24hour:Minute:Second(?:[.,]Microseconds)?(?:\s*Zone offset)?
    |  [0] {^LN-BEG}(?:DAY )?MON Day %k:Minute:Second(?:\.Microseconds)?(?: ExYear)?
    |  [0] {^LN-BEG}(?:DAY )?MON Day ExYear %k:Minute:Second(?:\.Microseconds)?
    |  [0] {^LN-BEG}Day(?P<_sep>[-/])Month(?P=_sep)(?:ExYear|ExYear2) %k:Minute:Second
    |  [0] {^LN-BEG}Day(?P<_sep>[-/])MON(?P=_sep)ExYear[ :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone offset)?
    |  [0] {^LN-BEG}Month/Day/ExYear:24hour:Minute:Second
    |  [0] {^LN-BEG}Month-Day-ExYear %k:Minute:Second(?:\.Microseconds)?
    |  [0] {^LN-BEG}Epoch
    |  [0] {^LN-BEG}ExYear2ExMonthExDay  ?24hour:Minute:Second
    |  [0] {^LN-BEG}MON Day, ExYear 12hour:Minute:Second AMPM
    |  [0] {^LN-BEG}ExYearExMonthExDay(?:T|  ?)Ex24hourExMinuteExSecond(?:[.,]Microseconds)?(?:\s*Zone offset)?
    |  [0] {^LN-BEG}(?:Zone name )?(?:DAY )?MON Day %k:Minute:Second(?:\.Microseconds)?(?: ExYear)?
    |  [0] {^LN-BEG}(?:Zone offset )?(?:DAY )?MON Day %k:Minute:Second(?:\.Microseconds)?(?: ExYear)?
    |  [0] {^LN-BEG}TAI64N
    |  [0] {^LN-BEG}24hour:Minute:Second
    |  [0] ^<Month/Day/ExYear2@24hour:Minute:Second>
    |  [0] ^MON-Day-ExYear2 %k:Minute:Second

   Lines: 161865 lines, 0 ignored, 364 matched, 161501 missed

    [processed in 361.65 sec]


    Missed line(s): too many to print.  Use --print-all-missed to print all 161501 lines

 

[Bitpalast]

When you see situations where attacks are coming from similar subnets, it is best to ban the whole subnet, for example 47.128.0.0/16. Like 47.128.59.193 for example is an IP of amazonaws.com, and their subnets are unfortunately known to cause significant load on servers.

Here we started banning networks since around April and have around 100 such subnets identified and banned for good on each and every host. We are also using scripts that remove single banned IPs from Fail2Ban once the whole subnet has been banned, because else you'll end up with tens of thousands of entries in iptables, which can cause significant issues with network package transport. It is important to keep the number of iptables entries as low as possible, so better group known attackers and block them altogether.

 

[JohnnyU]

Thank you for this advice, I will look into adding subnets to iptables.
In he last 48 hours I already have over 14,000 IPs blocked and the list is growing.

I have set the IP ban period to 600000 seconds so in a week the list would be cleared automatically or if I disable fail2ban after the attack is over

 

[Bitpalast]

Yeah, 14,000 is definitely too much. In general it works when the CPU is strong enough, but there will be moments where the NIC won't be able to process new packages, because in some seconds the Kernel might not be able to handle all the traffic. By my experience, if the list grows to more than 7,000 entries, things start getting problematic.