My own fail2ban banned me...

[Azurel]

What the f*ck happen here wrong? Thats now the second time. Used jail is "plesk-apache-badbot". Is here a stupid bug? I use chrome and firefox.

# Ban hosts which agent identifies spammer robots crawling the web
# for email addresses. The mail outputs are buffered.

[plesk-apache-badbot]
enabled = true
filter = apache-badbots
action = iptables-multiport[name=BadBots, port="http,https,7080,7081"]
logpath = /var/www/vhosts/system/*/logs/*access*log
/var/log/httpd/*access_log
maxretry = 1

/etc/fail2ban/filter.d/apache-badbots.conf

# Fail2Ban configuration file
#
# Regexp to catch known spambots and software alike. Please verify
# that it is your intent to block IPs which were driven by
# above mentioned bots.


[Definition]

badbotscustom = EmailCollector|WebEMailExtrac|TrackBack/1\.02|sogou music spider
badbots = Atomic_Email_Hunter/4\.0|atSpider/1\.0|autoemailspider|bwh3_user_agent|China Local Browse 2\.6|ContactBot/0\.2|ContentSmartz|DataCha0s/2\.0|DBrowse 1\.4b|DBrowse 1\.4d|Demo Bot DOT 16b|Demo Bot Z 16b|DSurf15a 01|DSurf15a 71|DSurf15a 81|DSurf15a VA|EBrowse 1\.4b|Educate Search VxB|EmailSiphon|EmailSpider|EmailWolf 1\.00|ESurf15a 15|ExtractorPro|Franklin Locator 1\.8|FSurf15a 01|Full Web Bot 0416B|Full Web Bot 0516B|Full Web Bot 2816B|Guestbook Auto Submitter|Industry Program 1\.0\.x|ISC Systems iRc Search 2\.1|IUPUI Research Bot v 1\.9a|LARBIN-EXPERIMENTAL \(efp@gmx\.net\)|LetsCrawl\.com/1\.0 +http\://letscrawl\.com/|Lincoln State Web Browser|LMQueueBot/0\.2|LWP\:\:Simple/5\.803|Mac Finder 1\.0\.xx|MFC Foundation Class Library 4\.0|Microsoft URL Control - 6\.00\.8xxx|Missauga Locate 1\.0\.0|Missigua Locator 1\.9|Missouri College Browse|Mizzu Labs 2\.2|Mo College 1\.9|MVAClient|Mozilla/2\.0 \(compatible; NEWT ActiveX; Win32\)|Mozilla/3\.0 \(compatible; Indy Library\)|Mozilla/3\.0 \(compatible; scan4mail \(advanced version\) http\://www\.peterspages\.net/?scan4mail\)|Mozilla/4\.0 \(compatible; Advanced Email Extractor v2\.xx\)|Mozilla/4\.0 \(compatible; Iplexx Spider/1\.0 http\://www\.iplexx\.at\)|Mozilla/4\.0 \(compatible; MSIE 5\.0; Windows NT; DigExt; DTS Agent|Mozilla/4\.0 efp@gmx\.net|Mozilla/5\.0 \(Version\: xxxx Type\:xx\)|NameOfAgent \(CMS Spider\)|NASA Search 1\.0|Nsauditor/1\.x|PBrowse 1\.4b|PEval 1\.4b|Poirot|Port Huron Labs|Production Bot 0116B|Production Bot 2016B|Production Bot DOT 3016B|Program Shareware 1\.0\.2|PSurf15a 11|PSurf15a 51|PSurf15a VA|psycheclone|RSurf15a 41|RSurf15a 51|RSurf15a 81|searchbot admin@google\.com|ShablastBot 1\.0|snap\.com beta crawler v0|Snapbot/1\.0|Snapbot/1\.0 \(Snap Shots, +http\://www\.snap\.com\)|sogou develop spider|Sogou Orion spider/3\.0\(+http\://www\.sogou\.com/docs/help/webmasters\.htm#07\)|sogou spider|Sogou web spider/3\.0\(+http\://www\.sogou\.com/docs/help/webmasters\.htm#07\)|sohu agent|SSurf15a 11 |TSurf15a 11|Under the Rainbow 2\.2|User-Agent\: Mozilla/4\.0 \(compatible; MSIE 6\.0; Windows NT 5\.1\)|VadixBot|WebVulnCrawl\.unknown/1\.0 libwww-perl/5\.803|Wells Search II|WEP Search 00

failregex = ^<HOST> -.*"(GET|POST).*HTTP.*"(?:%(badbots)s|%(badbotscustom)s)"$

ignoreregex =

# DEV Notes:
# List of bad bots fetched from http://www.user-agents.org
# Generated on Thu Nov 7 14:23:35 PST 2013 by files/gen_badbots.
#
# Author: Yaroslav Halchenko

My agent is for Chrome:

Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.89 Safari/537.36

and Firefox:

Mozilla/5.0 (Windows NT 6.3; WOW64; rv:37.0) Gecko/20100101 Firefox/37.0

 

[Sergey L]

Maybe you shall make your own IP trusted?

 

[Azurel]

Thats not possible. Its a dynamic IP. But why I'm banned? The filter looks for user agent of bots. Why its hit me with standard chrome/firefox agent?

Additional in /var/logs/message

Apr 22 09:08:23 mail fail2ban.actions[1853]: WARNING [plesk-apache-badbot] Ban **.**.**.**
Apr 22 09:08:23 mail fail2ban.actions[1853]: INFO [plesk-apache-badbot] **.**.**.** already banned

There is for days not any entry with this ip and i have no problems this day, till 09:08:23.

And... In (proxy)access_logs I only found 09:06 as last time stamp of me. I not understand this ban.

 

[UFHH01]

Hi Azurel,

And... In (proxy)access_logs I only found 09:06 as last time stamp of me. I not understand this ban.

The "apache-badbots" - filter has some regex - definitions, which can cause "normal" browsers to be banned as well. As you can see in your filter - configuration

|Mozilla/5\.0 \(Version\: xxxx Type\:xx\)|

... this specific part is the cause of you being hit by Fail2Ban. I suggest to delete "Mozilla/5\.0 \(Version\: xxxx Type\:xx\)|" in this definition, especially, if you define the very strict rule of "maxretry = 1" for apache-badbots. The other definitions are more precise and cause less or even none "false" banning of standard used browsers.

You were as well asking, if there is a "bug" in this configuration for apache-badbots - which is not the case. The standard used definition against these Romanian bots is still from the year 2007 and there were too less complaints about the mentioned definition, so that the developer of Fail2Ban didn't adjust it to a more precise one - the other reason is, that the site http://www.user-agents.org/ still list bots, which are not at all relevant anymore, because some user-agent-strings have been changed a long time ago.


Edit: Please be aware, that the symbol "|" is used to separate a definition. Keep an eye not delete a still necessary separator, when you delete one or more definitions!

 

[Azurel]

Thanks for your reply. The very strict rule coming from plesk, not me and I think plesk(plesk-apache-badbot) set the optimal setting for this filters?

But here is a major question. Why I get not banned each time I visit the website? I have fail2ban now enabled for weeks and two times now in the last 2 days I get banned. Thats strange!

 

[UFHH01]

Hi Azurel,

I can't answer your last question, without investigating your complete access - log - files and your complete fail2ban - logs. I could guess ( which I rarely do, because shoots out of the blue mostly never hit the real cause ), but I doubt that such a guess will help you to investigate your issue. I just try to suggest a work-around to solve your issue.

 

[LuigiMdg]

Sorry but after 8 years is this problem still present?
Because I didn't understand why it was giving me a connection_timed_out and then I found out I was banned..!
So far I've been very happy with Plesk, but if this issue occurs, I'll have to switch to another panel, I can't risk losing customers because my server is misconfigured.

 

[Peter Debik]

@LuigiMdg This issue has been solve years ago. On any new Plesk installation, the local IP address(es) are automatically whitelisted in Fail2Ban. If you are still on an old Plesk installation you need to add your IP addresses to the Fail2Ban whitelist manually.

 

[LuigiMdg]

@LuigiMdg This issue has been solve years ago. On any new Plesk installation, the local IP address(es) are automatically whitelisted in Fail2Ban. If you are still on an old Plesk installation you need to add your IP addresses to the Fail2Ban whitelist manually.

No, I have a fresh install, as I said I've been trying out Plesk, since yesterday to be exact.
I had manually whitelisted the IP, but that's not a solution since I don't have a static IP.

 

[Peter Debik]

Servers with dynamic IPs are not a supported scenario. Your server needs to have a static IP, let alone for the domains that are operated on that server.

 

[LuigiMdg]

Servers with dynamic IPs are not a supported scenario. Your server needs to have a static IP, let alone for the domains that are operated on that server.

But have you been drinking?
My IP is dynamic, not that of the domain, how could this?
It's me who bans F2B, not the server.
What's the point?

 

[Peter Debik]

So back to your original question I now understand that it is not the server's IP that is banned, but your ISP IP. This is the expected and desired behavior when access violations occur that are defined in the active Fail2Ban jails. You can check in /var/log/fail2ban.log which jail has banned your local IP and why it was banned.

 

[LuigiMdg]

So back to your original question I now understand that it is not the server's IP that is banned, but your ISP IP. This is the expected and desired behavior when access violations occur that are defined in the active Fail2Ban jails. You can check in /var/log/fail2ban.log which jail has banned your local IP and why it was banned.

For too many accesses on apache, but I don't find it correct, if a user opens more than one page is he banned?

 

[LuigiMdg]

Or for Mozilla 5.0..? Disabled

 

[Peter Debik]

Too many "client access denied" errors? Or too many requests in general? For the later there is no such Plesk jail and it would be interesting to know what the content of the jail is (the regexes etc.) that bans a user for too frequent requests.

 

[LuigiMdg]

I understand why it happened.. I use a spider to crawl the cache, I will insert a sleep between requests