Fail2ban problems in Plesk 12.0.18 update #68

[rbstern]

Not sure if the problem is related to the most recent update, but as best I can tell, it started sometime in the last couple of weeks.

I went into the fail2ban page of the panel to enable the ssh jail. I was surprised to find MANY more jails defined than I was used to seeing. Eighty jails in all. The new jails were not enabled, and I see no configuration files for any of them.

I looked at two other Plesk 12.0.18 #68 servers I manage, and those have 10 jails each. That's mystery #1.

When I tried to disable the ssh jail, I was greeted with the error message shown below. That's mystery #2.

Additional info: I had added a WordPress jail about six weeks ago. It's been working fine. That same Wordpress jail was also added to the other two servers.

The server in question is Centos 6.7. The other two servers are Centos 6.6 and 6.5.



[[errorJailNotDisabled]]
Unable to switch on the selected jails: f2bmng failed: ERROR NOK: ('Invalid command (no set action or not yet implemented)',)
ERROR NOK: ("invalid literal for int() with base 10: 'INFO'",)
ERROR NOK: ('Invalid command (no set action or not yet implemented)',)
ERROR NOK: ('Invalid command (no set action or not yet implemented)',)
ERROR NOK: (2, 'No such file or directory')
ERROR NOK: ('Invalid command (no set action or not yet implemented)',)
ERROR NOK: ('Invalid command (no set action or not yet implemented)',)
ERROR NOK: ('Invalid command (no set action or not yet implemented)',)
ERROR NOK: ('Invalid command (no set action or not yet implemented)',)
ERROR NOK: ('Invalid command (no set action or not yet implemented)',)
ERROR NOK: ('Invalid command (no set action or not yet implemented)',)
ERROR NOK: ('Invalid command (no set action or not yet implemented)',)
ERROR NOK: ('Invalid command (no set action or not yet implemented)',)
ERROR NOK: ('Invalid command (no set action or not yet implemented)',)
ERROR NOK: ('Invalid command (no set action or not yet implemented)',)
ERROR NOK: ('Invalid command (no set action or not yet implemented)',)
ERROR NOK: ('Invalid command (no set action or not yet implemented)',)
ERROR NOK: ('Invalid command (no set action or not yet implemented)',)
ERROR NOK: ('Invalid command (no set action or not yet implemented)',)
ERROR NOK: ('Invalid command (no set action or not yet implemented)',)
ERROR NOK: ('Invalid command (no set action or not yet implemented)',)
ERROR NOK: ('Invalid command (no set action or not yet implemented)',)
ERROR NOK: ('Invalid command (no set action or not yet implemented)',)
ERROR NOK: ('Invalid command (no set action or not yet implemented)',)
RROR NOK: (2, 'No such file or directory')
ERROR NOK: ('Invalid command (no set action or not yet implemented)',)
ERROR NOK: ('Invalid command (no set action or not yet implemented)',)
ERROR NOK: ('Invalid command (no set action or not yet implemented)',)
ERROR NOK: ('Invalid command (no set action or not yet implemented)',)
ERROR NOK: ('Invalid command (no set action or not yet implemented)',)
ERROR NOK: ('Invalid command (no set action or not yet implemented)',)
ERROR NOK: ('Invalid command (no set action or not yet implemented)',)
ERROR NOK: ('Invalid command (no set action or not yet implemented)',)
ERROR NOK: ('Invalid command (no set action or not yet implemented)',)
ERROR NOK: ('Invalid command (no set action or not yet implemented)',)
ERROR NOK: ('Invalid command (no set action or not yet implemented)',)
ERROR NOK: ('Invalid command (no set action or not yet implemented)',)
ERROR NOK: ('Invalid command (no set action or not yet implemented)',)
ERROR NOK: ('Invalid command (no set action or not yet implemented)',)
ERROR NOK: ('Invalid command (no set action or not yet implemented)',)
ERROR NOK: ('Invalid command (no set action or not yet implemented)',)
ERROR NOK: ('Invalid command (no set action or not yet implemented)',)
ERROR NOK: (2, 'No such file or directory')
ERROR NOK: ('Invalid command (no set action or not yet implemented)',)
ERROR NOK: ('Invalid command (no set action or not yet implemented)',)
ERROR NOK: ('Invalid command (no set action or not yet implemented)',)
ERROR NOK: ('Invalid command (no set action or not yet implemented)',)
ERROR NOK: ('Invalid command (no set action or not yet implemented)',)
ERROR NOK: ('Invalid command (no set action or not yet implemented)',)
ERROR NOK: ('Invalid command (no set action or not yet implemented)',)
ERROR NOK: ('Invalid command (no set action or not yet implemented)',)
ERROR NOK: ('Invalid command (no set action or not yet implemented)',)
ERROR NOK: ('Invalid command (no set action or not yet implemented)',)
ERROR NOK: ('Invalid command (no set action or not yet implemented)',)
ERROR NOK: ('Invalid command (no set action or not yet implemented)',)
ERROR NOK: ('Invalid command (no set action or not yet implemented)',)
ERROR NOK: ('Invalid command (no set action or not yet implemented)',)
ERROR NOK: ('Invalid command (no set action or not yet implemented)',)
ERROR NOK: ('Invalid command (no set action or not yet implemented)',)
ERROR NOK: ('Invalid command (no set action or not yet implemented)',)

[author note: above pattern repeats numerous times, but this form won't accept the post length]

ERROR:f2bmng:Command '['/usr/bin/fail2ban-client', 'reload']' returned non-zero exit status 255
ERROR:f2bmng:Failed to reload following jails due to errors in configuration
.

 

[rbstern]

Follow up. I just checked fail2ban versions, and discovered that the server where I am having the fail2ban problem is version 0.9.2, whereas the servers that are running correctly has fail2ban version 0.8.13.

I didn't install or update fail2ban on any of these servers. It came with via Plesk. Why different versions on Plesk servers with 12.0.18 update #68?

0.9.2 config file starts with a warning:

# WARNING: heavily refactored in 0.9.0 release. Please review and
# customize settings for your setup.

I suspect that's the basis of the problem.

 

[gbotica]

Hi,

I have had a similar situation too, this is to do with Fail2Ban being updated, and your Fail2Ban configuration requires updating, and the Fail2Ban server needs restarting.

I believe the difference is because CentOS 6.7 has fail2ban 0.9.2, whereas the older CentOS versions still have 0.8.x.

All those new Jails were installed by the 0.9.2 update.

All the errors you see are because fail2Ban needs a restart to understand the updated Fail2Ban config file.

/etc/init.d/fail2ban restart

You should check your /etc/fail2ban directory for any .rpmnew files and ensure that your config files are up to date.

Hope that helps.

 

[roadrash]

What gbotica is suggesting is correct. But just restarting fail2ban will most likely not help in all cases, depending on versions of OS.

Version 0.9.2 still ran ok on our CentOS 6.7 systems untill last week. The big problems began when Fail2ban upgraded to 0.9.3.
We use F2B with Iptables. There is the problem, because F2B 0.9.3 will no longer work correct with the IPtables version in CentOS 6.7

You should install the F2B version from the Plesk repo.
So if you have F2B installed from another repository, e.g. epel or rpmforge, then remove it.
Exclude fail2ban in all your repo's and then re-install it through Plesk Panel or autoinstaller and you're OK.

 

[gbotica]

Hi, Thanks for your comments @roadrash. Interesting to hear I'm not alone, I'm having serious issues with Fail2ban 0.9.3, CentOS 6.7 and Plesk 12.0.18 (thread here: http://talk.plesk.com/threads/plesk-12-0-18-update-68-and-fail2ban-0-9-3-problem.335183/).

When you say that "F2B 0.9.3 will no longer work with iptables in CentOS 6.7" is there a particular issue you are referring to? There was one breaking change mentioned in the F2B changelog (https://github.com/fail2ban/fail2ban/blob/master/ChangeLog) but I've implemented the fix as suggested, but it doesn't help. Excerpt:

* action.d/iptables-common.conf
- All calls to iptables command now use -w switch introduced in
iptables 1.4.20 (some distribution could have patched their
earlier base version as well) to provide this locking mechanism
useful under heavy load to avoid contesting on iptables calls.
If you need to disable, define 'action.d/iptables-common.local'
with empty value for 'lockingopt' in `[Init]` section.
​

Also, removing Fail2Ban and reinstalling from Plesk Panel / autoinstaller didn't work for me either as this installs 0.9.3 (which is when the problems started). I was thinking of trying to use YUM to downgrade back to 0.9.2, but I'm not sure if that's even possible?

 

[roadrash]

CentOS 6 will not go beyond version IPtables 1.4.7
Whereas F2B v0.9.3 uses functions that are only available from at least IPtables 1.4.20
I had the same errors as you see (the -w switch thing)

I first tried downgrading with yum. Wouldn't work unfortunately.

Best way to go (worked for ALL of our Centos 6 servers):

- stop fail2ban (service fail2ban stop), or disable the service through Plesk Panel
- backup your own custom f2b rules
- remove fail2ban: yum remove fail2ban (or: use the Plesk autoinstaller through Plesk Panel to remove the f2b addon)
- verify that all of the f2b things are gone.
- VERY important: make sure that you exclude fail2ban from all of your yum repos (in: /etc/yum.repos.d/ ).
- test it: yum install fail2ban (it should tell you that fail2ban cannot be found. That is OK! )
- Then login into the Plesk installer again and add the fail2ban again.

After that you can check that the Plesk-repo was used by following command:

yum info fail2ban
Output should contain following line: From repo: PLESK_12_5_30-dist
(if you are on Plesk 12.5)

Hope this helps you too.

 

[gbotica]

Thanks so much for your help, I will try that out and report back.

Weird thing is... Fail2Ban 0.9.3 itself (with CentOS 6.7, iptables 1.4.7) seems to be working fine (with the -w option disabled). F2B server runs fine, Jails load and successfully ban ips -- all seems to be working perfectly. (From my experience) it's only the Plesk Fail2Ban pages that are broken, in fact, I was completely unaware of any problems until I checked the Plesk Panel.

@roadrash, did you experience any other symptoms apart from the Plesk Panel 'Currently Banned IPs' and Jail detail view pages not loading?

 

[roadrash]

Hi gbotica,
I did not have any problems with Plesk 12.0.18 nor with 12.5.3 untill i just ran "yum update"
After that i noticed that the fail2ban panel in Plesk would not load.
So i started monitoring "/var/log/fail2ban.log" and noticed the "-w" errors.

 

[pleskpanel]

I can confirm that all of our CentOS 6.7 servers running the fail2ban 0.8.13-14052018 release are indeed running fine but interestingly enough, Plesk systems running 100 domain licenses only have the limited (10 or so) set of jails noted by rbstern while ten domain license Plesk installs have something closer to the 80-100 mark worth of non-active jails. This is on CentOS 6.7 (Final) Plesk version 12.0.18 Update #68.

What repositories are you using @roadrash?

 

[roadrash]

@pleskpanel
Repositories containing "fail2ban" are epel and rpmforge
Our servers now all have fail2ban from the Plesk repositories.
That is fail2ban version 0.9.2

 

[gbotica]

CentOS 6 will not go beyond version IPtables 1.4.7
Whereas F2B v0.9.3 uses functions that are only available from at least IPtables 1.4.20
I had the same errors as you see (the -w switch thing)

I first tried downgrading with yum. Wouldn't work unfortunately.

Best way to go (worked for ALL of our Centos 6 servers):

- stop fail2ban (service fail2ban stop), or disable the service through Plesk Panel
- backup your own custom f2b rules
- remove fail2ban: yum remove fail2ban (or: use the Plesk autoinstaller through Plesk Panel to remove the f2b addon)
- verify that all of the f2b things are gone.
- VERY important: make sure that you exclude fail2ban from all of your yum repos (in: /etc/yum.repos.d/ ).
- test it: yum install fail2ban (it should tell you that fail2ban cannot be found. That is OK! )
- Then login into the Plesk installer again and add the fail2ban again.

After that you can check that the Plesk-repo was used by following command:

yum info fail2ban
Output should contain following line: From repo: PLESK_12_5_30-dist
(if you are on Plesk 12.5)

Hope this helps you too.

I've done this on one of my servers and Fail2Ban 0.8.13 was installed from repo: PSA_12_0_18-dist.

I haven't reinstalled my custom jails and filters etc, but so far so good. Shame to have to downgrade back to 0.8.x but I guess it's not really such a big deal. I presume I could manually install 0.9.2 and then I would be back to where I started before it broke.

I'll report back if I run into any further trouble.

Thanks very much to all contributors for their help with this.

 

[gbotica]

I decided to try getting back to Fail2Ban 0.9.2 as this has numerous improvements compared to 0.8.13 and has worked fine with Plesk 12.0.18.

So, what I did was:

1. removing fail2ban completely
2. excluded fail2ban from the EPEL repo
3. installing fail2ban 0.9.2 manually (https://github.com/fail2ban/fail2ban/archive/0.9.2.tar.gz)

So far all working fine.

Thanks again for all your help.

 

[trialotto]

@rbstern, @roadrash, @gbotica and @pleskpanel,

The issue here is that you use the extra packages associated with EPEL.

Note that without EPEL activated, you (all) should have fail2ban version 0.8.13 on Plesk 12.0.18 and fail2ban version 0.9.2 on Plesk 12.5.30.

The Odin provided fail2ban version is somewhat different from the "standard" fail2ban versions (that can be obtained from the fail2ban project pages).

For the sake of clarity, let´s refer to the before mentioned "standard" fail2ban versions as "project versions".

It is not a mystery that Plesk´s fail2ban has a limited number of jails: only the project versions have 100 plus jails (and this is not dependent on the specific project version, i.e. both the 0.8.x and 0.9.x project versions have a huge number of jails, as opposed to the Plesk fail2ban version).

As such, gbotica is partly right: the new jails come with the 0.9.x version, but only with the 0.9.x project version.

It is not the case that a Plesk update or micro-update has caused the issues or the deviant behavior of fail2ban: it is simply the presence of EPEL, in particular the extra packages that cause the fail2ban packages to be upgraded to a newer project version, mostly the latest 0.9.x version.

Note that the activation of EPEL can also result in an automatic installation of non-stable project versions of fail2ban.

It is not the case that the current issue is dependent on a subversion of an OS (in this case, old versus new versions of CentOS): every subversion with EPEL activated will show very similar behavior, in the sense that a 0.9.x project version can be installed when triggering some kind of updating process.

Furthermore, it is not the case that an EPEL based OR a Plesk installer based OR a Plesk micro-update based upgrade of fail2ban will require a restart of fail2ban (i.e. regardless of the version used): the erratic behavior of not properly (re- or) starting is the result of having a 0.9.x project version not being compatible with other packages.

As such, roadrash hints the proper solution to the current issue: use the Odin provided fail2ban package, i.e. use the proper Plesk repos.

As such and more important, gbotica and roadrash hint to the cause of the error messages: the fail2ban 0.9.3 project version (installed at EPEL based upgrade on Centos 6.7) is not compatible with specific iptable versions (and/or the common work-around to this issue is not implemented).

In conclusion, just use the Plesk fail2ban packages.

In addition, I noticed that various fail2ban versions are used on various versions of Plesk and as a final note, I must state: use the intended version of fail2ban, being 0.8.13 for Plesk 12.0.18 and 0.9.2 for Plesk 12.5.30.

In theory and in practice, one could opt for the 0.9.2 version in Plesk 12.0.18 (this has to be done manually and can require some tweaking, depending on the OS), but there is no real advantage of using the 0.9.x versions, in stead of the 0.8.x versions, in Plesk 12.0.18 (in fact, it is better to upgrade Plesk to 12.5.30).

Kind regards.....

 

[gbotica]

Hi,
Thanks for your comments. I need the EPEL repo on my servers for other packages that I run. I see now that the best thing to do would have been to exclude Fail2Ban from EPEL intially, which would have ensured that a Plesk compatible version of Fai2Ban was installed. Having said that, I've been running Fail2Ban 0.9.2 for quite some time with no issues at all.

 

[trialotto]

@gbotica,

It is not surprising that you do not encounter problems with fail2ban 0.9.2 packages, since the Odin provided 0.9.2 version and the 0.9.2 project version should work properly.

It does not matter whether you are using Plesk 12.0.18 or Plesk 12.5.30.

Note that I did some extensive testing with various fail2ban packages and from that perspective, I can state:

- the 0.9.1 and 0.9.2 versions are not that stable as suggested and, in addition, are less stable than the 0.8.13 or 0.8.14 versions,
- there is a huge difference between the 0.9.x and the 0.8.x versions, primarily related to introduction of persistent IP bans by means of a database,
- the 0.9.x are still in "development", in the sense that bugfixes are still being introduced,

and so on.

The "new business model" of version 0.9.x is somewhat strange, in the sense that persistent bans are now possible by means of

a) iptables, as such a proven concept of a very reliable and persistent method of IP banning,

b) usage of existing blacklists, i.e. (web)services that provide access to various IPs that should be banned,

c) the fail2ban database, as such not "good practice" (permanent bans should be in iptables AND the fail2ban environment becomes more vulnerable to program failure)

d) the fail2ban connections to and/or integration with the before mentioned "blacklists", as such already available in the 0.8.x versions and improved in the 0.9.x versions,

and it should be clear by now that the new functions in fail2ban 0.9.x versions are not really adding value to the existing software stacks.


In essence, one should better use the "bare" fail2ban 0.8.x version and, in addition, use iptables and some familiar and proven "blacklists".

Really, simple 0.8.x fail2ban, some firewall rules (i.e. iptables) and -optionally- the use of Cloudflare (which also allows a domain based IP ban, instead of only a server-wide IP ban, as is possible with fail2ban 0.9.x versions) is the only combination of software that makes any sense.

In my humble opinion, fail2ban is very valuable and practical, but the 0.9.x versions are "too much of the same thing" and, moreover, associated with a huge performance impact.

The before mentioned performance impact can become very relevant: a server with certain combinations of hard- and software can become (very) vulnerable and even unprotected, when confronted with certain attacks, causing system overloads and/or fail2ban-server process overloads (i.e. errors, primarily with the database).

Furthermore, be aware that even a properly functioning fail2ban will result in more than one paradox:

- fail2ban only detects "failed attempts" and does not make a distinction between malicious traffic and good traffic (everybody can make a mistake, this should not result in a ban),
- fail2ban is just as good as the definition of filters, actions and rules (and that takes more time and knowledge than one can imagine),
- fail2ban has a huge impact on iptables without any form of efficiency (it is easier to block IP ranges, instead of block individual IPs or whitelist individual IPs or IP ranges),
- fail2ban uses the method of "banning IPs afterwards", whilst blocking IPs beforehand is more practical and more efficient,
- fail2ban does NOT detect malicious traffic being succesfull (i.e. malicious traffic can enter the server, even if fail2ban is setup properly),

and that is not even the full summary.

In short, my advice: use 0.8.13 or 0.8.14 versions of fail2ban AND do not rely solely on fail2ban to block malicious traffic from certain IPs.

Kind regards.....

 

[gbotica]

@trialotto

Thanks so much for your detailed summary of your testing and experience with Plesk and Fail2Ban -- that's really helpful to know.

However, it does beg the question: If Fail2ban 0.8.x is more stable and performant that 0.9.x, why is Plesk 12.5 using it?

 

[trialotto]

@gbotica,

To answer the question

If Fail2ban 0.8.x is more stable and performant that 0.9.x, why is Plesk 12.5 using it?

I must admit and state: I am not sure, in the sense that the answer to that question is not known to me.

In essence, using 0.9.x versions is an investment in the future: the development of the 0.8.x branch will cease and fail2ban 0.8.14 then remains the last version with the "old structure".

As a result, it is more easy to adopt to the "new structure", introduced by the 0.9.x versions, with the additional advantage that persistent IP bans are possible, which persistent IP bans are not present in the 0.8.x versions, but are highly demanded by many Plesk customers, including those customers using Plesk 12.0.18.

In my humble opinion, it was a good strategy by Odin to (only) adopt the 0.9.x versions in Plesk 12.5.30, as opposed to upgrading fail2ban in both the new and old version of Plesk.

Still, the major pitfall of fail2ban 0.9.x is that it actually requires a database to create persistent IP bans, even though iptables is being used.

Hence, the lack of persistent iptables rulesets, created by fail2ban, is resolved by introducing the "database method" and this is odd, very odd: any iptable ruleset created by a firewall application is persistent (that is, in most cases), whilst fail2ban chooses and requires to realise persistency in a different way (note, it is a matter of choice AND necessity).

In fact, in the case of fail2ban, the matter of necessity is the result of the structure of the original core code, that has not really been adjusted, but augmented each time.

Even though fail2ban is properly maintained, its core code is a little bit messy, amongst others due to the emphasis on development of additional features (as often demanded by the public and/or as deemed valuable by the fail2ban project developers) and, in a sense, less emphasis on development or even improvement of core functionality.

In short, one should be aware of some of the pitfalls of any version of fail2ban AND, in addition, be aware that solely relying on fail2ban is "bad practice".

However, fail2ban is one tool (of many) that can or should be used to keep a server safe and it is logical to adopt the latest version of fail2ban.

I certainly can advice you to use fail2ban (any stable version) AND to take some time to configure it properly AND to create some work-arounds for some of the pitfalls of fail2ban.

For instance, when using 0.9.x versions, try to disable the database and try to use a firewall based ruleset for persistent IP banning (note that fail2ban 0.9.x can fail, i.e. freeze and shutdown, when overloaded by many requests, for instance in the case of distributed hack attempts).

As another illustration, the fail2ban config files do allow for a lot of optional functionalities (for instance, reporting banned IPs to specific blacklists) and it often is best to disable most or all of the optional functionalities, in order to improve performance and/or to prevent that fail2ban fails.

As a final illustration, be aware that many of the tools on Plesk are achieving more or less the same: for instance, restricting SSH access to the IP of the sysadmin with one firewall based ruleset makes ssh jails in fail2ban obsolete (and iptbales becomes very inefficient, if the ssh jail remains activated). Also note that extensions or external applications, such as Cloudflare´s "stop the hacker", can do the same as fail2ban, but do it much better (again, specific fail2ban jails will become obsolete and will make iptables inefficient).

In general, one should keep constantly in mind that

- fail2ban is not intended to gather banned IP addresses (one of the reasons why the "database method" for IP bans is inherently odd)
- fail2ban´s sole purpose should be to keep malicious attempts out of the server (as opposed to malicious traffic)
- firewall based rulesets in iptables should be used to keep malicious traffic out of the server
- firewall based rulesets in iptables should be used to block most or all IPs, with the exception of a limited amount of allowed IPs
- good firewall based rulesets often leave port 80 and ftp and mail related ports to be monitored by fail2ban (and anything else would result in waste of resources by fail2ban)

and hence, fail2ban can help in

- preventing the first parts of any kind of distributed attack (again, note that fail2ban is not able to prevent a full-blown distributed attack)
- preventing hack attempts with respect to ftp based services (note that using TLS and/or passive ports is a better option, in comparison to using fail2ban)
- preventing hack attempts with respect to mail based services

and therefore, it is probably clear that one cannot overexagerate the role, functionality and added value of fail2ban.

In essence, I believe that the biggest pitfall of fail2ban and/or firewalls is that these security measures do not account for succesfull hack attempts.

For that reason, I would also advice to make use of a tool like AIDE (advanced intrusion detection) or something similar.

Hope the above helps.

Kind regards.....

 

[gbotica]

Hello, the issue was reported to development team with ID: PPPM-3464

Plesk 12.0.18 Update #68 and Fail2ban 0.9.3 problem