Fail2Ban issues

[Xavier12]

Hi guys,

We're having several issues with Fail2ban in Plesk. When adding an ip address to "Add A trusted ip address", the system hangs for more than a minute until it saves the new trusted ip address. At times it does not work and fail2ban gives an error which gives an option to restart the fail2ban server.

Please advise what would be needed from my end and what exact logs should I provide. Thanks guys

 

[IgorG]

Please advise what would be needed from my end and what exact logs should I provide.

All necessary information about Plesk troubleshooting is here:

[IMPORTANT]: READ THIS FIRST TO FIND QUICK SOLUTION FOR THE EXISTING ISSUE/QUESTION

 

[Xavier12]

Hi Igor,

Thanks for the update. Here are the system details:

‪CloudLinux Server 6.7
12.5.30 Update #13
Postfix, dovecot

As far as Fail2Ban, I am not seeing any mentions of it within the link.

Please advise, thanks.

 

[IgorG]

Have you tried any mentioned steps for troubleshooting/fixing this issue? Any detail, results?
Sorry, but "It doesn't work! Please help!" is really not enough.

 

[Xavier12]

Hi Igor,

Thanks for reaching back. Still not resolved, there is a delay when adding ips to the trusted ip address list. Tried to reinstall and restart the service, still not resolved. If anything comes up, will be sure to post the result/resolution

 

[Ripshod]

If it's a configuration problem then simply reinstalling won't work unless you make sure the following directories are deleted:

These are directories as placed in ubuntu, they may be elsewere (I'm not familiar with Cloudlinx)
/var/lib/fail2ban
/run/fail2ban
/etc/fail2ban.

Delete any stragglers after uninstalling or the old files/database may cause problems again.

 

[Xavier12]

If it's a configuration problem then simply reinstalling won't work unless you make sure the following directories are deleted:

These are directories as placed in ubuntu, they may be elsewere (I'm not familiar with Cloudlinx)
/var/lib/fail2ban
/run/fail2ban
/etc/fail2ban.

Delete any stragglers after uninstalling or the old files/database may cause problems again.

Hi Ripshod,

Thanks so much for the response. Makes sense. How can I backup the settings as far as the existing trusted ip addresses and ban options configuration? This way I don't have to reconfigure them all over again. I do not have too many Trusted ip addresses. I'd say about 10. Is it worth transferring settings or should i just delete all and start from scratch by entering them manually again?

Please advise, thanks!

 

[Ripshod]

Whitelisted IPs are stored in /etc/fail2ban/jail.local

Would make sense just to download that file, uninstall fail2ban, clean up any errant files, install fail2ban, stop fail2ban, upload the file (overwriting preserves permissions) and finally start fail2ban.

I did find (at a later date) that setting fail2ban to keep the database in memory is advantageous when you have a lot of domains on the server. That setting is in /etc/fail2ban/fail2ban.conf. Since setting to 'memory' I've not had a failure. Only downside is a server reboot will rebuild the database from scratch.

One other trick I learned is not to disable fail2ban with the normal tick box (as we naturally assume) but to switch it of from the 'Banned IP Addresses' tab.

The settings themselves are spread over 4 files, and as at least one of these files has other config info would be best to just lose those settings and redo them. I don't think you'll need to make the settings again, But you can download /etc/fail2ban/jail.local /etc/fail2ban/jail.conf /etc/fail2ban/fail2ban.conf and /etc/fail2ban/jail.d/plesk.conf as a backup. Only problem there is if you change things on the server you'd need to download those files again.

 

[Xavier12]

Hi Ripshod,

Thanks for the response! What do you mean by "Only downside is a server reboot will rebuild the database from scratch". Does this mean all configuration is lost for trusted ips, jails etc?

Please advise, thanks.

 

[Xavier12]

So, after testing, it seems that fail2ban slows down after enabling a few Jail options.. the ones that slow the system down the most when adding and removing trusted ips are:

plesk-apache badbot
plesk-wordpress

Ive deleted all configuration files and have completely reinstalled. Still the issues continues to exist.

Please advise, thanks.

 

[Ripshod]

So, after testing, it seems that fail2ban slows down after enabling a few Jail options

Stop the fail2ban service. Search /etc/fail2ban/fail2ban.conf for 'dbfile ='. Make that line read dbfile = memory then restart fail2ban.

What do you mean by "Only downside is a server reboot will rebuild the database from scratch"

Settings and whitelisted IPs are stored in the various conf files. I believe he database stores the current bans as well as the relevant counts. So any existing bans would be reset - but there's no real harm done there, we don't reboot every day after all and the database only has a 24hr life. before IPs are purged.
If you still have slowdowns it may be a good idea to set log rotation to daily, then the fail2ban code has an easier time parsing the log files.

 

[trialotto]

So, after testing, it seems that fail2ban slows down after enabling a few Jail options.. the ones that slow the system down the most when adding and removing trusted ips are:

plesk-apache badbot
plesk-wordpress

Ive deleted all configuration files and have completely reinstalled. Still the issues continues to exist.

Please advise, thanks.

@Xavier12,

It is rather logical that apache and WordPress related jails are a little bit troublesome: after all, these are the jails handling bot crawling.

A simple improvement of Fail2Ban performance would be involving:

a) blocking some IPs of specific crawlers, such as the Baidu spider (and such alike), via the Plesk Firewall (or any other firewall), (and)

b) adjusting robots.txt files.

The above measures will remove some of the "stress" on Fail2Ban, try it and have a look at the results afterwards.....

Also note that you can have a look at the apache (and other) logs, to identify some of the (truely) malicious IPs and simply block them with a firewall.

Hope the above helps!

Regards.....

 

[trialotto]

@Rhipsod (and @Xavier),

With respect to the suggestion of using the ":memory:" option, this generally is a bad idea, for many reasons.

The most important reason is that the usage of this option does not tackle the root cause of the problem: a heavy workload for Fail2Ban.

Using the ":memory:" option will only shift the problem to memory and that can be or become very problematic.

For a more general outline and/or explanation, have a view at my post here: http://talk.plesk.com/threads/disk-utilization-increased-after-upgrade-from-12-0.336450/#post-795059

Hope this helps a bit.

Regards....

 

[Ripshod]

setting fail2ban to keep the database in memory is advantageous when you have a lot of domains on the server

I found by experience that using a database file with a lot of domains, and therefore a lot of logs, causes quite a load in the form of disk reads/writes, slowing the whole server. It can actually get to such a load that fail2ban can stall (fail) - defeats the object. Reducing the database reads/writes does seem to reduce this load significantly.

However thanks for pulling me on that point, it needed explaining a little more. Great input as always

 

[Xavier12]

Hey Trialotto,

Thanks for the in-depth response. Here are the two issues:

a) blocking some IPs of specific crawlers, such as the Baidu spider (and such alike), via the Plesk Firewall (or any other firewall)
We are a little concerned about enabling the Plesk firewall due to previously bad experience. After we enabled it in the past, many ports were blocked/modified and several services somehow stopped working or functioning correct. Because of this previous issue, we ended up having to re-install a whole new server and plesk and refrain from using the Plesk firewall option.

b) adjusting robots.txt files.
Since we have a good amount of customers, it would be difficult to create a robots.txt files for each website installation.

We've tried to re-install and are now having issues with enabling and re-enabling fail2ban.. especially when the Jails are enabled. There is an option to permanently ban the unwanted ip addresses by sending them over to iptables permanently, but fail2ban performance just isnt performing as it should. Even checked apache logs and nginx logs, nothing displayed. There was even a point where I visited the plesk homescreen and revisited fail2ban in tools & settings and it hanged until an nginx 504 error appeared. Rebooted the server several times, was even forced to reboot the server in order to uninstall fail2ban because Plesk updates and upgrades option was hanging without fully uninstalling. Very weird where these issues are coming from. I can't be the only one having these kind of issues.. lol

Using the following:
Cloudlinux 6.7
Plesk 12.5.30 Update #20
(even disabled cloudlinux features: lvemanager and cagefs)
Mariadb 10.0.23

Please advise, thanks guys!

 

[trialotto]

@Xavier12

First of all, I am rather suprised that you choose the Cloudlinux system, without having lvemanager enabled. In such a situation, you would be better off with Ubuntu.

Nevertheless, some general remarks and tips and for the sake of convenience, I will quote some parts of your response.

We are a little concerned about enabling the Plesk firewall due to previously bad experience. After we enabled it in the past, many ports were blocked/modified and several services somehow stopped working or functioning correct. Because of this previous issue, we ended up having to re-install a whole new server and plesk and refrain from using the Plesk firewall option.

You should not worry about that.

In general, the Plesk firewall is a GUI for iptables management and it certainly will not cause specific services to stop working and/or working improper.

It can be the case that you blocked some ports, not allowing specific services to connect through the firewall.

If you want to test Plesk firewall, just enable it with all rulesets set on "allow", implying that all traffic is getting through AND also implying that, in the very rare case that your system stops working properly, it indeed was the Plesk firewall causing your system to stop functioning.

I am pretty sure that you can enable the Plesk firewall without any problems for the CloudLinux system.

Make sure that you deactivate Fail2Ban service before you enable the Plesk Firewall test, otherwise your "test results" would be biased.

Since we have a good amount of customers, it would be difficult to create a robots.txt files for each website installation.

Not really.

In essence, the robots.txt can be identical on all sites, certainly if the purpose is to block specific (bad) crawlers.

The one (and only) robots.txt file can be copied to each of the httpdocs directories for (all) the domains you have. That seems to be a lot of work, but it really isn´t.

On the other hand, why bother if one has Fail2Ban?

Ehm, there we are again: we want Fail2Ban, but Fail2Ban has some peculiar issues.

Certainly, you are right when stating

There is an option to permanently ban the unwanted ip addresses by sending them over to iptables permanently, but fail2ban performance just isnt performing as it should.

and I must say that the bold part of the quote always has been true, in the sense that not every function in Fail2Ban functions properly.

Nevertheless, most of the "alleged Fail2Ban issues" have something to do with bad configuration.

And, more important, configuration improvement can also be used to "work-around" the problems with specific Fail2Ban functions.


Problem is..... while writing this post, I became a little bit unfocused, since I thought of some specific Fail2Ban setup that we tested in a far past.

That specific setup can be a full resolution of your (and other´s) problem with Fail2Ban.

I will first have a look at that, is that a good suggestion?

Regards....

 

[Xavier12]

Hi Trialotto,

Thanks you for the detailed breakdown. I am actually using LVEmanager and CageFS. What i meant was that I even tried to temporarily disable those to see if the issue was the cloudlinux features, and they arent.

Will take a shot and consider the provided information and reach back if I can resolve anything. Thnx!

 

[Xavier12]

Tested the Fail2ban issue alone, here is what the log stated. Not sure if the issues are happening because of iptables. After turning on fail2ban, then turning it off, the system became unstable within the fail2ban menu. Could not stop the service via command line or Plesk. Only way to solve the issue is to reboot the system and uninstall fail2ban.

016-01-27 01:28:16,695 fail2ban.jail [17747]: INFO Jail 'ssh' started
2016-01-27 01:28:16,718 fail2ban.jail [17747]: INFO Jail 'recidive' started
2016-01-27 01:28:16,741 fail2ban.jail [17747]: INFO Jail 'plesk-proftpd' started
2016-01-27 01:28:16,768 fail2ban.jail [17747]: INFO Jail 'plesk-postfix' started
2016-01-27 01:28:16,793 fail2ban.jail [17747]: INFO Jail 'plesk-dovecot' started
2016-01-27 01:28:16,817 fail2ban.jail [17747]: INFO Jail 'plesk-roundcube' started
2016-01-27 01:28:16,866 fail2ban.jail [17747]: INFO Jail 'plesk-apache' started
2016-01-27 01:28:16,947 fail2ban.jail [17747]: INFO Jail 'plesk-apache-badbot' started
2016-01-27 01:28:16,987 fail2ban.jail [17747]: INFO Jail 'plesk-panel' started
2016-01-27 01:28:17,026 fail2ban.jail [17747]: INFO Jail 'plesk-wordpress' started
2016-01-27 01:28:26,401 fail2ban.server [17747]: INFO Stopping all jails
2016-01-27 01:28:27,147 fail2ban.action [17747]: ERROR iptables -D INPUT -p tcp -j f2b-default
iptables -F f2b-default
iptables -X f2b-default -- stdout: ''
2016-01-27 01:28:27,150 fail2ban.action [17747]: ERROR iptables -D INPUT -p tcp -j f2b-default
iptables -F f2b-default
iptables -X f2b-default -- stderr: 'iptables: Too many links.\n'
2016-01-27 01:28:27,151 fail2ban.action [17747]: ERROR iptables -D INPUT -p tcp -j f2b-default
iptables -F f2b-default
iptables -X f2b-default -- returned 1
2016-01-27 01:28:27,152 fail2ban.actions [17747]: ERROR Failed to stop jail 'plesk-apache-badbot' action 'iptables-allports': Error stopping action
2016-01-27 01:28:27,414 fail2ban.jail [17747]: INFO Jail 'plesk-apache-badbot' stopped
2016-01-27 01:28:28,475 fail2ban.jail [17747]: INFO Jail 'recidive' stopped
2016-01-27 01:28:29,574 fail2ban.jail [17747]: INFO Jail 'plesk-roundcube' stopped
2016-01-27 01:28:29,999 fail2ban.jail [17747]: INFO Jail 'plesk-panel' stopped
2016-01-27 01:28:31,033 fail2ban.jail [17747]: INFO Jail 'plesk-apache' stopped
2016-01-27 01:28:32,057 fail2ban.jail [17747]: INFO Jail 'plesk-dovecot' stopped
2016-01-27 01:28:32,982 fail2ban.jail [17747]: INFO Jail 'ssh' stopped
2016-01-27 01:28:34,014 fail2ban.jail [17747]: INFO Jail 'plesk-postfix' stopped
2016-01-27 01:28:34,343 fail2ban.jail [17747]: INFO Jail 'plesk-proftpd' stopped

 

[Ripshod]

Do you have Plesk Firewall enabled? I believe you need this.
Are you using a different firewall?

 

[trialotto]

@Rhipsod,

I must add to my previous response and your previous statement

I found by experience that using a database file with a lot of domains, and therefore a lot of logs, causes quite a load in the form of disk reads/writes, slowing the whole server. It can actually get to such a load that fail2ban can stall (fail) - defeats the object. Reducing the database reads/writes does seem to reduce this load significantly.

and admit that I have had a glance at some of the jails, filters and actions.

The conclusion was (amongst others) that

a) there is some unnecessary overhead, due to definition of logs that have to be scanned:

- they are not aligned and/or do not correspond with the actual logs to be scanned,
- unnecessary usage of wildcards in the log file definitions,

and it still has to be determined whether this impacts resource usage to some significant degree,

b) there seems to be a rather unlogical structure of jails, in the sense that

- jails are scanning log files in duplicate (i.e. the same log files are scanned multiple times, by various jails),
- specific jails will not do what they aim to do or are intended to do,

and, even though the above can be convenient for some purposes, it can cause performance issues,

c) one or two additional actions, augmented with a simple rewrite of some jails (i.e. applying the new action), could drastically improve Fail2Ban performance (tested, check),

d) an improvement of the Plesk Panel´s Fail2Ban implementation would be desirable: configurable settings for the findtime variable,

e) Fail2Ban performance improvements are in essence "consequences of bad practice": one of the pitfalls of relying on Fail2Ban is to be found in the peculiar disadvantage of Fail2Ban, being that Fail2Ban can only take action when some malicious traffic or request has been logged. That is, action after harm has been done. On a particular test server, a firewall has been used to render Fail2Ban effectively useless: no IP banning by Fail2Ban, since the firewall prevents (almost) all undesirable traffic.

In short, given all of the above and point e in particular, it leaves me again to consider that the firewall would be more important, hence questioning Fail2Ban´s functions.

Nevertheless, I will continue to test the new actions (see point c) and report back to this topic thread. That can take some time though.

Regards...