• Please be aware: Kaspersky Anti-Virus has been deprecated
    With the upgrade to Plesk Obsidian 18.0.64, "Kaspersky Anti-Virus for Servers" will be automatically removed from the servers it is installed on. We recommend that you migrate to Sophos Anti-Virus for Servers.
  • The Horde webmail has been deprecated. Its complete removal is scheduled for April 2025. For details and recommended actions, see the Feature and Deprecation Plan.

Question New bad bot fail2ban filter: will overwrite customization?

J

JVG

Basic Pleskian
Server operating system version
Debian 12
Plesk version and microupdate number
Plesk Obsidian 18.0.62
Ouch, yes, it seems at least one of the recent updates has replaced the file. The new version is "ok", but it seems someone used the version form my article How to Avoid High CPU Load & Block Bad Bots with Plesk - Plesk . This is valid, but we, too, have customizations in addition to what's shown in the article that seem to having been lost, and I am now in the process of updating the files here again with our custom version. I didn't notice earlier, because we are also collecting and banning subnets so that most attacks by the customizations didn't occur any longer.

@Sebahat.hadzhi That would be something to consider to not to overwrite custom /etc/fail2ban/filter.d/apache-badbots.conf, but only overwrite this if it's an "original" file version.
 
Thank you for your input. I have opened an internal case for the described behavior for our team to investigate and apply the necessary changes to preserve the configuration, if possible. I will update you as soon as there are more details.
 
Hello, guys. Our engineers reviewed the matter and their response is that the file is not brought by Plesk. It is coming from the actual fail2ban package from OS repositories. Therefore, its OS package is what overrides it upon an update, technically making this behavior non Plesk-related.

With that said, I have taken the liberty to create a UserVoice request and I would like to encourage you to vote for it if you want the config to be integrated into the Plesk core in future. At this point, the alternative we can suggest is to consider creating a backup copy of the config file prior to an update.
 
  • Like
Reactions: JVG
Sebahat.hadzhi said:
Hello, guys. Our engineers reviewed the matter and their response is that the file is not brought by Plesk. It is coming from the actual fail2ban package from OS repositories. Therefore, its OS package is what overrides it upon an update, technically making this behavior non Plesk-related.

With that said, I have taken the liberty to create a UserVoice request and I would like to encourage you to vote for it if you want the config to be integrated into the Plesk core in future. At this point, the alternative we can suggest is to consider creating a backup copy of the config file prior to an update.
Click to expand...

Hi, no problem for me at all making a copy before updating, I just wanted to know if I had to.
 
Sebahat.hadzhi said:
Hello, guys. Our engineers reviewed the matter and their response is that the file is not brought by Plesk.
Click to expand...
Nope, that's not possible, because the "thesis..." bot in the custom section is an example that is not part of the official file.
It's not a big problem, the file is fine in general. There are some things that can be improved such as the regex that @Kaspar@Plesk once suggested ( failregex = ^<HOST> -[^"]*"(?:GET|POST|HEAD) \/.* HTTP\/\d(?:\.\d+)" \d+ \d+ "[^"]*" "[^"]*(?:%(badbots)s|%(badbotscustom)s)[^"]*"$ ) which is an excellent improvement.

Let's vote for the Uservoice entry. Fail2Ban is becoming more important every day. Radware had published in a recent report that the number of attacks against servers quadrupled since December 2023, so we can expect that investing into fast and sufficient ban rules will be beneficial for all users.
 
Sebahat.hadzhi said:
Hello, guys. Our engineers reviewed the matter and their response is that the file is not brought by Plesk. It is coming from the actual fail2ban package from OS repositories. Therefore, its OS package is what overrides it upon an update, technically making this behavior non Plesk-related.
Click to expand...
Are you sure? I mean, the official bad bot filter from fail2ban hasn't been updated for over 7 years. Why would it suddenly be overwitten from the OS repository? That would pretty much render the custom bad bot filter from Plesk pointless, if it can be overwritten with the default bad bot filter from the OS repo at any time.

Bitpalast said:
Ouch, yes, it seems at least one of the recent updates has replaced the file.
Click to expand...
It's odd that this happend, because to my knowledge a modified filter file should not get overwritten by the Plesk update. I have a (heavily) modified bad bot filter too, but on each of my servers with Plesk .63 my modified bad bot filter was retained and instead a apache-badbots.conf.rpmnew file got created with the new Plesk bad bot filter.

That being said and taking into account that apparently filters can be overwritten from OS repositories, the best practice seems to be to save all fail2ban related modification to a .local file, instead of to the default .conf files. At least for files (jails and filters) that exist in the public repositories.
Source: Proper fail2ban configuration
 
Last edited:
I can positively say that the update overwrites a customized apache-badbots.conf file. When I realized I updated all these files yesterday with our custom setup. On one server that updated to 18.0.63 #3 today I again see this content:

Code: 
# Fail2Ban configuration file
#
# Regexp to catch known spambots and software alike. Please verify
# that it is your intent to block IPs which were driven by
# above mentioned bots.


[Definition]

badbotscustom = thesis-research-bot
badbots = GPTBot|AmazonBot|Bytespider|Bytedance|fidget-spinner-bot|EmailCollector|WebEMailExtrac|TrackBack/1\.02|sogou music spider|seocompany|LieBaoFast|SEOkicks|Uptimebot|Cliqzbot|ssearch_bot|domaincrawler|AhrefsBot|spot|DigExt|Sogou|MegaIndex\.ru|majestic12|80legs|SISTRIX|HTTrack|Semrush|MJ12|Ezooms|CCBot|TalkTalk|Ahrefs|BLEXBot|Atomic_Email_Hunter/4\.0|atSpider/1\.0|autoemailspider|bwh3_user_agent|China Local Browse 2\.6|ContactBot/0\.2|ContentSmartz|DataCha0s/2\.0|DBrowse 1\.4b|DBrowse 1\.4d|Demo Bot DOT 16b|Demo Bot Z 16b|DSurf15a 01|DSurf15a 71|DSurf15a 81|DSurf15a VA|EBrowse 1\.4b|Educate Search VxB|EmailSiphon|EmailSpider|EmailWolf 1\.00|ESurf15a 15|ExtractorPro|Franklin Locator 1\.8|FSurf15a 01|Full Web Bot 0416B|Full Web Bot 0516B|Full Web Bot 2816B|Guestbook Auto Submitter|Industry Program 1\.0\.x|ISC Systems iRc Search 2\.1|IUPUI Research Bot v 1\.9a|LARBIN-EXPERIMENTAL \(efp@gmx\.net\)|LetsCrawl\.com/1\.0 \+http\://letscrawl\.com/|Lincoln State Web Browser|LMQueueBot/0\.2|LWP\:\:Simple/5\.803|Mac Finder 1\.0\.xx|MFC Foundation Class Library 4\.0|Microsoft URL Control - 6\.00\.8xxx|Missauga Locate 1\.0\.0|Missigua Locator 1\.9|Missouri College Browse|Mizzu Labs 2\.2|Mo College 1\.9|MVAClient|Mozilla/2\.0 \(compatible; NEWT ActiveX; Win32\)|Mozilla/3\.0 \(compatible; Indy Library\)|Mozilla/3\.0 \(compatible; scan4mail \(advanced version\) http\://www\.peterspages\.net/?scan4mail\)|Mozilla/4\.0 \(compatible; Advanced Email Extractor v2\.xx\)|Mozilla/4\.0 \(compatible; Iplexx Spider/1\.0 http\://www\.iplexx\.at\)|Mozilla/4\.0 \(compatible; MSIE 5\.0; Windows NT; DigExt; DTS Agent|Mozilla/4\.0 efp@gmx\.net|Mozilla/5\.0 \(Version\: xxxx Type\:xx\)|NameOfAgent \(CMS Spider\)|NASA Search 1\.0|Nsauditor/1\.x|PBrowse 1\.4b|PEval 1\.4b|Poirot|Port Huron Labs|Production Bot 0116B|Production Bot 2016B|Production Bot DOT 3016B|Program Shareware 1\.0\.2|PSurf15a 11|PSurf15a 51|PSurf15a VA|psycheclone|RSurf15a 41|RSurf15a 51|RSurf15a 81|searchbot admin@google\.com|ShablastBot 1\.0|snap\.com beta crawler v0|Snapbot/1\.0|Snapbot/1\.0 \(Snap Shots&#44; \+http\://www\.snap\.com\)|sogou develop spider|Sogou Orion spider/3\.0\(\+http\://www\.sogou\.com/docs/help/webmasters\.htm#07\)|sogou spider|Sogou web spider/3\.0\(\+http\://www\.sogou\.com/docs/help/webmasters\.htm#07\)|sohu agent|SSurf15a 11 |TSurf15a 11|Under the Rainbow 2\.2|User-Agent\: Mozilla/4\.0 \(compatible; MSIE 6\.0; Windows NT 5\.1\)|VadixBot|WebVulnCrawl\.unknown/1\.0 libwww-perl/5\.803|Wells Search II|WEP Search 00

failregex = ^<HOST> -.*"(GET|POST|HEAD).*HTTP.*"(?:%(badbots)s|%(badbotscustom)s)"$

ignoreregex =

datepattern = ^[^\[]*\[({DATE})
              {^LN-BEG}

This is the content from the Plesk article.
 
Sebahat.hadzhi said:
Hey, guys. I have further discussed the case with our team and the config was updated through an upstream request submitted by one of our team members:
Click to expand...
Thank your for disusing this with the team Sebahat. In all honesty this is a bit confusing, because none of those changes from the Pull Request you linked to where accepted and merged into main/master branch of fail2ban. Besides the changes to the bad bot filter in the Pull Request are different from the recent changes made to the bad bot filter for the .63 Plesk release. Which means that, as far as I can see at least, there could have been no way for the bad bot filter to be overwritten/updated from the upstream OS repo (with the same content from the .63 Plesk release). So there must be something else going on here.

Just to clearify, I am not affected by this as on none of my servers the bad bot filter got overwritten. Instead the updated bad bot filter from Plesk got saved to the apache-badbots.conf.rpmnew file, which as far as I remember is the intended behavior. So I am not sure what's causing the overwriting issue @Bitpalast is encountering.
 
Wow, actually I can confirm there is an issue. I decided to manually update another server to .63 just now. And the apache-badbots.conf file did get overwritten, even tough it was modified before. @Sebahat.hadzhi I think this needs to be look at.

STR (tested on Alma 8):
1) Install server with Plesk .62
2) Modify apache-badbots.conf
3) Update to Plesk .63
4) See apache-badbots.conf got overwritten with the version from Plesk

Expected behavior: apache-badbots.conf file does not get overwritten when it has been modified by user.
 
Last edited:
It seems that it gets overwritten with a "wrong" timestamp, too. I am sure I had an updated apache-badbots.conf from Aug 27 on one machine where today an update was done. After the update to 18.0.63 #3 the timestamp was Aug 19, and the content was again the version from the article I once wrote for Plesk.
 
Hi everyone,

Can't tell for sure is it expected or not for their file in github to remain "updated 7 years ago", yet you can download the package, for example, from here: https://ubuntu.pkgs.org/22.04/ubuntu-universe-amd64/fail2ban_0.11.2-6_all.deb.html and unpack(`ar x *.deb` && `tar -xvf data.tar.zst` on ubuntu) it to review it's contents. To be specific, this: etc/fail2ban/filter.d/apache-badbots.conf and you'll find it's content different from what is posted on github

Technically, I would call it quite expected for packages to overwrite their files.

In this particular edge-case scenario, Plesk does provide a re-packed version of fail2ban. Even though the file is shipped as is eventually, as @Sebahat.hadzhi stated, I've registered it as #PPPM-14581, with expectations like:

Either of:
- Modifications made by users are preserved
- There’s a GUI(/CLI) method to manage the list of bad bots and config is generated by Plesk
 
Never mind, we've now added a surveillance routine for the /etc/fail2ban/filter.d/apache-badbots.conf file. If it is replaced or modified, we'll automatically replace it with our master copy again and reload Fail2Ban. Problem solved.
 
You must log in or register to reply here.

Similar threads

T
Question Fail2ban trying to setup a rule to ban ai bots
Replies
0
Views
372
Tomsohner
T
Azurel
Question Fail2ban filter apache-badbots incomplete?
Replies
1
Views
2K
DataPacket
DataPacket
Back
Top