Question Sudden drop in Fail2Ban banned IPs — anyone else seeing this?

[Azurel]

Server operating system version

AlmaLinux 9.8

Plesk version and microupdate number

18.0.81#2

Hi everyone,

for years, my total number of banned IPs hovered around 3,000. Over the past few days, I’ve noticed that the number of banned IPs has dropped to around 700. That’s unusually low, and quite a surprise. Has anyone else noticed something similar? Fewer attacks would be welcome, but such a sharp drop after years of fairly consistent numbers seems unusual. Could this be a bug, or have the default ban times been shortened recently? fail2ban is running and banning

 

[Sebahat.hadzhi]

Hi, @Azurel . As far as I am aware, no changes related to fail2ban were recently performed. I also can't find recent bug reports with similar symptoms. Could you please check if you have some failures in the logs:

journalctl -u fail2ban

 

[Bitpalast]

@Azurel Number of attacks went up sharply this year, so it is very unlikely that less attacks are coming in to your server. But maybe you are experiencing a change in the attack scheme that F2B cannot catch without modifications: Many large scale attackers are now using distributed attacks across many websites on the same server "at once", either from the same IP or sometimes from a handful of IPs. These occur at almost the same moment across many websites on the same system, hence go unnoticed by F2B. They are not actually brute-forcing, but rather checking for .env files, backup files, .gz, .zip etc. - anything that promises to deliver backup copies of access credentials. So there might be a shift of computing power on hackers' side away from useless attempts of brute-forcing to these multiple bad bot attacks. I call them "bot burst", maybe that can make the standard name for the scheme some day ...

 

[King555]

I'm also seeing an increase of attacks and bad bot traffic this year, like @Bitpalast says, so you @Azurel might just have luck.

 

[Azurel]

The number of banned IPs has dropped even further and is now down to around 400. It would be very helpful if the Jails overview /admin/server-protection/jail-list showed the number of banned IPs per jail. A chart per Jail-Line covering the last six months or so would be even better. You could then see at a glance which jail or filter is behind such a drastic change. As it stands, I can't trace this back, and I've been searching for a cause for quite a while without finding anything, because I have no clue which jail is responsible for this.

@Sebahat.hadzhi
# journalctl -u fail2ban

Sep 30 03:40:06 xxx fail2ban-server[448141]: Server ready
Oct 02 07:17:36 xxx systemd[1]: Stopping Fail2Ban Service...
Oct 02 07:18:03 xxx fail2ban-client[984781]: Shutdown successful
Oct 02 07:18:03 xxx systemd[1]: fail2ban.service: Deactivated succe>
Oct 02 07:18:03 xxx systemd[1]: Stopped Fail2Ban Service.
Oct 02 07:18:03 xxx systemd[1]: fail2ban.service: Consumed 11h 20mi>
-- Boot 0ca002e13fb74958b868ae2d85eaea92 --
Oct 02 07:20:04 xxx systemd[1]: Starting Fail2Ban Service...
Oct 02 07:20:04 xxx systemd[1]: Started Fail2Ban Service.
Oct 02 07:20:05 xxx fail2ban-server[1376]: Server ready

 

[Kaspar]

The number of banned IPs has dropped even further and is now down to around 400. It would be very helpful if the Jails overview /admin/server-protection/jail-list showed the number of banned IPs per jail.

Clicking each jail shows you the list of banned IP's for that particular jail. It is a bit inconvenient that you have to click trough each jail to see the number of banned IPs, but the number are there. Not sure if you are aware already, I wanted to point this out in case you aren't.

Similarly via command line you can run:

sudo fail2ban-client status <JAIL-NAME>

A chart per Jail-Line covering the last six months or so would be even better. You could then see at a glance which jail or filter is behind such a drastic change. As it stands, I can't trace this back, and I've been searching for a cause for quite a while without finding anything, because I have no clue which jail is responsible for this.

Sounds like a great use case for AI. Just upload the fail2ban logs to your favorite AI tool and ask it to run an analysis on the logs. If you want a comprehensive analysis upload your access logs too (maybe anonymize them first).

 

[Azurel]

Sounds like a great use case for AI.

You're so right! Thanks for the reminder. I just did that, and the AI detected exactly that problem in the log files and was also able to identify the jail that's different and creates an Excel file. That's was a big help.