Poor fail2ban implementation

[Christian_Heutger]

After damaging my installation with different recommendations here, I was able to get fail2ban running on my CentOS 7 system just by removing the EPEL fail2ban and manually installing the ones from Parallels (although outdated):

#./plesk-installer
#rpm -qa | grep "fail2ban"
#rpm -e --nodeps fail2ban-sendmail-0.9-9.el7.noarch
#rpm -e --nodeps fail2ban-firewalld-0.9-9.el7.noarch
#rpm -e --nodeps fail2ban-systemd-0.9-9.el7.noarch
#rpm -e --nodeps fail2ban-server-0.9-9.el7.noarch
#rpm -e --nodeps fail2ban-0.9-9.el7.noarch
#wget http://autoinstall.pp.parallels.com...n/fail2ban-0.8.13-centos7.14071718.noarch.rpm
#yum install fail2ban-0.8.13-centos7.14071718.noarch.rpm

However, as this isn't the current version with todays updating of CentOS 7 to 7.1.1503 I wasn't able to start fail2ban any more as the selinux policies seems to be updated (based on the actual versions), so fail2ban-client was not able to create the sock and pid files any more. Adding rules to SElinux failed also as fail2ban_client_t seems not to be registered. Working with Parallels Plesk's fail2ban is now only possible any more if doing a CLI

#setenforce 0
#systemctl start fail2ban
#setenforce 1

But that can't be the idea of an admin panel? Need a solution!

 

[trialotto]

@Christian_Heutger,

The Fail2Ban 0.9.x versions are development releases, the 0.8.13 and 0.8.14 (latest) are stable releases and therefore the Fail2Ban versions to be installed.

It is very likely that either your previous 0.9.x Fail2Ban installation hinders the proper installation of version 0.8.13, OR you should simply install (or re-install) the fail2ban module with the autoinstaller from Plesk (instead of wgetting the rpm package and installing it).

In my opinion, it should suffice to

1 - run ..../admin/sbin/autoinstaller
2 - select number 8 (Fail2Ban authentication failure monitor) in the list, OR deselect number 8 (if it is already present)
3 - proceed with installation
4 - if Fail2Ban has been "deselected" in step 2, just run the autoinstaller again to install Fail2Ban

and this way, you should have a clean install of Fail2Ban, without the problems you did mention.

If the problems persist, please let me know (I will try to replicate the whole error and find a solution).

Kind regards...

 

[Christian_Heutger]

I tried the reinstallation recently which result in damaging the Plesk installation completely. However as I use EPEL still the packages of 0.9 been choosen.

However, I retried but same behaviour. I need to mention, that fail2ban recently worked well, but with CentOS update to the recent release fail2ban broke. Seems that the selinux policy has been adjusted and as 0.8 is nothing, which is supported for CentOS 7 by RedHat it fails. I would believe, that version 0.9 which is official provides for CentOS 7 via EPEL would have no issues.

 

[trialotto]

I tried the reinstallation recently which result in damaging the Plesk installation completely. However as I use EPEL still the packages of 0.9 been choosen.

However, I retried but same behaviour. I need to mention, that fail2ban recently worked well, but with CentOS update to the recent release fail2ban broke. Seems that the selinux policy has been adjusted and as 0.8 is nothing, which is supported for CentOS 7 by RedHat it fails. I would believe, that version 0.9 which is official provides for CentOS 7 via EPEL would have no issues.

Christian,

The Plesk installation will not be broken by an install and/or re-install of Fail2Ban module, the Plesk core packages should still be working properly.

It can be the case that an unproper uninstall of Fail2Ban 0.9.x does make it impossible to install the 0.8.x versions of Fail2Ban.

It can also be the case that the rpm-based install of Fail2Ban (as executed by you) is different from the Plesk based installation of Fail2Ban.

However, even with the above mentioned "additional points of attention", the issue should be primarily due to selinux settings.

One should check for selinux errors by using the command: journalctl -lfu fail2ban.

Check for messages such as: " SELinux is preventing /usr/bin/python2.7 from ..... [removed for simplicity] .... Do allow this access for now by executing:
# grep fail2ban-server /var/log/audit/audit.log | audit2allow -M mypol
# semodule -i mypol.pp "

Just follow up the advice and execute the commands, but be aware that you have to restart Fail2Ban and (again) check for any error messages. It is an iterated process.

Anyway, before you are doing anything, follow the steps below:

a) remove (i.e. uninstall) all Fail2Ban packages and associated files (also make sure that /etc/fail2ban directory is deleted),

b) install the Fail2Ban module (note: this is a 0.8.x version) via the Plesk autoinstaller (i.e. run the autoinstaller from the command line),

and, afterwards, check for selinux related errors and repair those selinux related errors by the above mentioned commands.

Kind regards....

 

[Christian_Heutger]

Hi,

still does not work:

-- Logs begin at Sa 2015-04-04 19:46:01 CEST. --
Apr 06 00:33:37 vm1.heutger.net fail2ban-client[19411]: ERROR Directory /var/run/fail2ban exists but not accessible for writing
Apr 06 00:33:37 vm1.heutger.net systemd[1]: fail2ban.service: control process exited, code=exited status=255
Apr 06 00:33:37 vm1.heutger.net systemd[1]: Failed to start Fail2ban Service.
Apr 06 00:33:37 vm1.heutger.net systemd[1]: Unit fail2ban.service entered failed state.
Apr 06 00:33:37 vm1.heutger.net systemd[1]: fail2ban.service holdoff time over, scheduling restart.
Apr 06 00:33:37 vm1.heutger.net systemd[1]: Stopping Fail2ban Service...
Apr 06 00:33:37 vm1.heutger.net systemd[1]: Starting Fail2ban Service...
Apr 06 00:33:37 vm1.heutger.net systemd[1]: fail2ban.service start request repeated too quickly, refusing to start.
Apr 06 00:33:37 vm1.heutger.net systemd[1]: Failed to start Fail2ban Service.
Apr 06 00:33:37 vm1.heutger.net systemd[1]: Unit fail2ban.service entered failed state.

[root@vm1 ~]# cat mypol.te
module mypol 1.0;
require {
type var_run_t;
type fail2ban_t;
class sock_file { create unlink getattr };
}
#============= fail2ban_t ==============
#!!!! This avc is allowed in the current policy
allow fail2ban_t var_run_t:sock_file { create unlink getattr };

It's always the same issue.

Regards,
Christian

 

[trialotto]

@Christian_Heutger,

Can you confirm that you are using CentOS 7 and Fail2Ban version 0.8.13 (as provided by Plesk)?

Any specific version numbers and/or information about the server would be appreciated.

I will try to replicate the error. That (and some testing) will take some time, but I will respond afterwards to your post(s).

Kind regards....

 

[trialotto]

@Christian_Heutger,

I have setup a VM with CentOS 7, made a clean install of Plesk and installed Fail2Ban separately (after installation of Plesk).

I cannot replicate the error, it should be working in your server.

The above implies that an error should not be caused by selinux settings, hence implying that it is very likely that

a) (either) the fail2ban 0.9.x version has not been properly and/or completely removed,

b) (or) the fail2ban 0.8.13 version is not installed properly (since it is not likely that the fail2ban official version differs from the version provided by the Parallels repo)

and hence I would recommend you to make some (intensive) effort to remove fail2ban and all remaining files, in order to make a clean new install (with autoinstaller).

Kind regards.....

 

[Christian_Heutger]

plesk-fail2ban-configurator-12.0.18-cos7.build1200140807.16.noarch
fail2ban-0.8.13-centos7.14071718.noarch

CentOS Linux release 7.1.1503 (Core)
Derived from Red Hat Enterprise Linux 7.1 (Source)
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"
CentOS Linux release 7.1.1503 (Core)
CentOS Linux release 7.1.1503 (Core)
cpe:/o:centos:centos:7

The reason why your installation works is that you miss the EPEL repository, but I need it for different reasons, most because of image optimization tools. Once adding the EPEL repository fail2ban is fetched from the EPEL repository and this is version 0.9.x. If doing an uninstall and installing via Plesk instead of manually installing the RPMs (however that's that e.g. Virtuozzo does if you work with containers, they don't run the autoinstaller, they install the RPMs or more detailed link the symlinks to the installed RPMs in the vz repository), always the EPEL version is fetched. I have no idea on how to force the version been installed by adjusting yum, when the installation is done via autoinstaller. I'm able to prevent yum to update fail2ban with the EPEL repository, but as I don't see any parallels plesk repository, I can't add and also can't install via yum (which maybe a reason for selinux to fail after updating CentOS to release 7.1.1503 just recently, before it was 7.0.1406, for sure always patched with latest updates, but that didn't break fail2ban, just the last bigger update to the new release 7.1.1503 included some extra updates like selinux and breaked fail2ban, which after manual install recently worked.

But I'm sure it's about selinux after looking at different (more destroying) helps like in this forum recommending an iptables -F which result in kicking me out of the system completely, I came across, that the issue is strange and all strange issues are in conjunction with selinux (e.g. there are selinux bugs which prevent OpenVPN from being used by default, need special adjustment (Fix selinux context of files: restorecon -Rv /etc/openvpn). Also looking the journal, it exactly states:

Apr 06 13:16:30 vm1.heutger.net fail2ban-client[129006]: ERROR Directory /var/run/fail2ban exists but not accessible for writing
Apr 06 13:16:30 vm1.heutger.net systemd[1]: fail2ban.service: control process exited, code=exited status=255
Apr 06 13:16:30 vm1.heutger.net systemd[1]: Failed to start Fail2ban Service.
Apr 06 13:16:30 vm1.heutger.net systemd[1]: Unit fail2ban.service entered failed state.

but the directly is accessible for writing, I also changed the rights to 777, no changes. However, looking at audit.log with audit2allow, I directly see:

type=AVC msg=audit(1428318989.451:8071): avc: denied { write } for pid=128986 comm="fail2ban-client" name="fail2ban" dev="tmpfs" ino=14979 scontext=system_u:system_r:fail2ban_client_t:s0 tcontext=system_ubject_r:var_run_t:s0 tclass=dir
Was caused by:
Missing type enforcement (TE) allow rule.
You can use audit2allow to generate a loadable module to allow this access.

However, creating a SElinux rule therefore, recently result in an error, that the type fail2ban_client_t does not exist. Using setenforce 0 afterwards everything worked well, but that couldn't be the solution.

However, now it seemed to work out (not sure why) as I first did the steps you recommended and afterwards did the steps I recently did:

grep fail2ban-client /var/log/audit/audit.log | audit2allow -M mypol2
semodule -i mypol2.pp

systemctl start fail2ban failed again

grep fail2ban-client /var/log/audit/audit.log | audit2allow -M mypol3
semodule -i mypol3.pp

systemctl start fail2ban started successfully

However, it looks-a-like Plesk is not really SElinux-compatible as consulting the audit.log there are many more problems in conjunction with Plesk which seems to be the reason why some additional strange topics occur, e.g. xferlog is always empty:

type=AVC msg=audit(1425783060.027:122912): avc: denied { read write } for pid=26750 comm="httpd" path="/var/asl/data/updates-data" dev="sda3" ino=2578583 scontext=system_u:system_r:httpd_t:s0-s0:c0.c1023 tcontext=system_ubject_r:var_t:s0 tclass=file

type=AVC msg=audit(1425786803.191:123052): avc: denied { write } for pid=32605 comm="py-limit-out" path="/usr/local/psa/var/psasem.sem" dev="sda3" ino=2149093341 scontext=system_u:system_r:sendmail_t:s0 tcontext=unconfined_ubject_r:var_t:s0 tclass=file

type=AVC msg=audit(1425786803.256:123053): avc: denied { write } for pid=32606 comm="check-quota" path="/usr/local/psa/var/psasem.sem" dev="sda3" ino=2149093341 scontext=system_u:system_r:sendmail_t:s0 tcontext=unconfined_ubject_r:var_t:s0 tclass=file

type=AVC msg=audit(1425786803.270:123054): avc: denied { write } for pid=32609 comm="sendmail.postfi" path="/usr/local/psa/var/psasem.sem" dev="sda3" ino=2149093341 scontext=system_u:system_r:sendmail_t:s0 tcontext=unconfined_ubject_r:var_t:s0 tclass=file

type=AVC msg=audit(1425823993.114:124420): avc: denied { open } for pid=81221 comm="in.proftpd" path="/var/log/plesk/xferlog" dev="sda3" ino=3221527233 scontext=system_u:system_r:ftpd_t:s0-s0:c0.c1023 tcontext=system_ubject_r:cron_log_t:s0 tclass=file

type=AVC msg=audit(1428140424.860:656): avc: denied { read } for pid=4018 comm="smtp" name="resolv.conf" dev="sda3" ino=1073846856 scontext=system_u:system_rostfix_smtp_t:s0 tcontext=system_ubject_r:httpd_sys_content_t:s0 tclass=file (which also started with the update)

type=AVC msg=audit(1428133956.001:288): avc: denied { open } for pid=2391 comm="httpd" path="/var/log/modsec_audit.log" dev="sda3" ino=2149026881 scontext=system_u:system_r:httpd_t:s0 tcontext=system_ubject_r:var_log_t:s0 tclass=file

Using an admin panel which finally does not work with the base system (which now has SElinux enabled by default) is really sad. Always adjusting SElinux can't be the solution.

For anyone following this thread, fail2ban can be fixed by:

module mypol 1.0;
require {
type var_run_t;
type fail2ban_t;
class sock_file { create unlink getattr };
}

#============= fail2ban_t ==============
#!!!! This avc is allowed in the current policy
allow fail2ban_t var_run_t:sock_file { create unlink getattr };

module mypol2 1.0;
require {
type var_run_t;
type fail2ban_client_t;
class dir write;
}

#============= fail2ban_client_t ==============
allow fail2ban_client_t var_run_t:dir write;

module mypol3 1.0;
require {
type var_run_t;
type fail2ban_client_t;
class sock_file write;
class dir write;
}

#============= fail2ban_client_t ==============
#!!!! This avc is allowed in the current policy
allow fail2ban_client_t var_run_t:dir write;
allow fail2ban_client_t var_run_t:sock_file write;

I will now try to solve all the other policy issues.

 

[trialotto]

@Christian_Heutger,

First of all, thanks for the elaborate answer.

For your information: an EPEL based clean installation has been used, but I did not do an upgrade from version 7.0.1406.

The issue on hand, as encountered by you, can now be narrowed down to two causes, being

- selinux related upgrade issues, and/or
- plesk not supporting specific EPEL packages,

and the solution(s) for the above are relevant for future Plesk releases.

I will do the "upgrade scenario" testing, because (in my opinion) the current issues should not occur when using Plesk with EPEL.

Hopefully this replaces the mixed conclusions up till now, since the selinux adjustments can (indeed) not be the desired solution.

Kind regards....

 

[Christian_Heutger]

I'm no expert to selinux but my further investigation looks like:

- Plesk conflicts with EPEL (as mentioned by you) as it expects no EPEL installed and therefore expects packages not being available in the original repositories been taken from Parallels repository. However, that's a bit ignoring real-life environments, as well as Parallels allows/recommends often different repositories, e.g. atomic, but EPEL seems not to be one of them. However, I would prefer Plesk would use the lastest fail2ban version, as been somehow "official" provided via EPEL.
- Plesk conflicts with selinux, with first installation it seems to adjust all the settings for selinux environment working well with Plesk. However, it is usual, best practice, recommended and required to update also the base system (beside Plesk) and it seems like the policy-updates damage the Plesk adjustments, as Plesk seems not to use yum therefore. I would appreciate to update the system also via Plesk (e.g. many appliances do), so Plesk could keep their own selinux adjustments working. However, it seems, as too often in my time doing Hosting with Virtuozzo and Plesk, that the panel is not really tested with environments promoted as compatible with Plesk, as the selinux enablement by default with CentOS 7 seems not to be recognized.

However for the upgrade testing, I'm afraid, there are two topics:

fail2ban conflicting with EPEL
upgrades conflicting with selinux

They are somehow different, but just showed up as of the upgrade selinux issues once again. For the time being my manual "reinstallation" of fail2ban worked out well and now works out well again with selinux adjustments. I will do an
audit2allow -a -w
audit2allow -a -M mypolicy
semodule -i mypolicy.pp

from time to time to keep my system "running".

 

[trialotto]

@Christan_Heutger,

I did some additional testing and the results become even more confusing, but some final and simple solution should be present.

However, we (first) do have to go back to the basics.

From a very fast test, I can conclude that

1. the version of CentOS (being either 7.0.1406 or 7.1.1503) does not or should not matter,
2. the use of EPEL itself does not necessarily cause the problems encountered,
3. the Fail2Ban version (being either 0.8.13 or 0.9.1) does not or should not matter,
4. as such, an upgrade of Fail2ban itself does not necessarily cause the problems encountered (i.e. these problems are invariant to upgrades),

and the problems encountered seem to arise from the installation of Fail2Ban.

To be more precise, the installation of Fail2Ban with EPEL AND/OR, equivalently, the use of EPEL and the upgrade command with associated installation of Fail2Ban (version 0.9.1) will cause the automatic start of Fail2Ban (server).

The problem is not that Plesk will not recognize the (new) version of Fail2Ban, given the fact that Plesk can use the EPEL based installation of Fail2Ban.

The problem is in fact that Plesk does not recognize the Fail2Ban (server) process, as started after the EPEL based installation.

The factual cause: the EPEL based solution starts Fail2Ban with the -x option.

The solution is simple:

a) apply the method presented in http://kb.odin.com/en/120960, but this solution is very likely to fail, (OR)

b) go to the Plesk panel, to "Server Management", "Tools & Settings", select "IP Address Banning (Fail2Ban)" and do the following:

- go to Jails tab,
- select "Entries per page: All",
- select all jails and click on "Switch Off",
- go to "Settings" tab,
- select (i.e. select checkbox) "Enable intrusion detection"
- press Enter (or "OK")

and Fail2Ban will function properly (irregardless of the version of Fail2Ban).

In short, the problems you encountered were caused by a conflict between server processes (given that one Fail2Ban process was already started).

Note that it therefore is not and/or should not be necessary to change selinux settings and/or to change Fail2Ban versions and/or uinstall/re-install Fail2Ban.

Hope the above helps....

Kind regards....

PS As a personal note, it would adviceable in your personal case to remove all previous installations of Fail2Ban and, as desired by you, to use the EPEL based installation of Fail2Ban packages, followed by the steps in before mentioned point b). This way, you should have a "more clean" CentOs system.

 

[Christian_Heutger]

Hi,

so plesk-fail2ban (as read before at http://talk.plesk.com/threads/plesk12-centos-7-fail2ban-0-9-9-el7-error.314386/) does NOT depend on fail2ban-0.8? With first installation with fail2ban-0.9 I got jails, which are based on 0.9 and fail with 0.8. I was required to clean up the jails.

Good recommendation, many thanks. However, there seem to be bugs with Plesk and selinux as by migrating from a CentOS 6 server to CentOS 7 Wordpress stopped to work with FTP and I tried many things, creating upgrade folder, hardcoded FTP details, everything did not work out, I was required to change to file system based updates. Now I encountered, it's the same issue: selinux prevent proftpd from accessing anything. proftpd came with Plesk.

Regards,
Christian

 

[trialotto]

@christian,

so plesk-fail2ban (as read before at http://talk.plesk.com/threads/plesk12-centos-7-fail2ban-0-9-9-el7-error.314386/) does NOT depend on fail2ban-0.8?

No, the Fail2Ban version 0.9.1 seems to be supported by Plesk GUI, with some minor caveats (see below).

With first installation with fail2ban-0.9 I got jails, which are based on 0.9 and fail with 0.8. I was required to clean up the jails.

This is true, one of the caveats is a huge number of "jails" that have to be cleaned up. For the sake of convenience, one can also leave them "as is" (i.e. inactive).

However, there seem to be bugs with Plesk and selinux as by migrating from a CentOS 6 server to CentOS 7 Wordpress stopped to work with FTP and I tried many things, creating upgrade folder, hardcoded FTP details, everything did not work out, I was required to change to file system based updates. Now I encountered, it's the same issue: selinux prevent proftpd from accessing anything. proftpd came with Plesk

I am just brainstorming, can it not be the case that selinux does not "like" the "older" versions of packages, as used and installed by Plesk?

Note that I use "older" to indicate that you use EPEL, which is very likely to require different package versions.

Or is it something similar to the Fail2Ban issue, that is, EPEL based installation somehow creates a conflict between processes?

To be honest, taking an educated guess, it should not be the case that selinux and/or Plesk is bothered by the proftpd version.

In the case of proftpd, there is some difficulty in the fact that the start or restart of the ftp server is dissimilar to the (re)start of Fail2Ban.

At this moment, I do not have the time to replicate and test your case, but if you have some patience, I can see what I can do.

By the way, a personal question, why do you choose the 7.1.1503 release, if the 7.0.1406 version is better suited for Plesk?

Kind regards....

 

[Christian_Heutger]

Hi,

proftpd never had issues, it got installed by Plesk, but it also get updated by yum update. So I'm afraid, it's nothing about an older version, as /usr/local/psa/var/log/xferlog always should be writeable (but isn't because of selinux policies). So I'm afraid Plesk installation failed to set xferlog to be writeable.

Just fail2ban failed with 7.1.1503 update, however, it's no distribution upgrade, it's just using the standard yum update I use all the time to keep my packages up2date. I usually expect that as best practice to keep the installation up2date and secure (e.g. fixing security bugs). I never encountered such issues before, it was the first time. But I encountered many strange issues with selinux and systemd. However I expect from Plesk to be able to manage that, if Plesk is stating it's CentOS 7 compatible, it also should be.

As I now add everything via semanage, take your time for testing. ;-)

Regards,
Christian

 

[IgorG]

Guys,

I can say only one thing. As far as I know Fail2ban 0.9 version in gamma stage now (http://www.fail2ban.org/wiki/index.php/Main_Page). Stable is 0.8.x version. We can't use and support gamma third-party software in Plesk.

 

[Christian_Heutger]

Somehow it's what official comes from the vendor (RedHat) as part of their official recommended and supported repository for RHEL and derivates. However, the issue are many selinux inconsistences, some like proftpd not working by default are not in conjunction with any unsupported version, e.g. /usr/local/psa/var/log/xferlog location seems not to be selinux compatible. However selinux is and should be enabled by default with RHEL 7 and derivates (official supported) and should be used to be in compliance with different IT security standards.

 

[Noturns]

Is fail2ban the only decent anti hacker thats freeware?

 

[trialotto]

@Noturns

No, but Fail2Ban has one of the best sets of options and, more important, is (on the one hand) easy to use and relatively "worry free" and (on the other hand) some major improvements can be expected in the near future, such as a permanent IP block (instead of the current temporary IP ban).

Kind regards.....

 

[Noturns]

@trialotto do you perhaps know a workaround to simulate a permanent IP ban/block ?

 

[trialotto]

@Noturns,

Yes and no.

Yes, I am aware of multiple theoretical solutions to automatically set a permanent IP block.

No, you should not want to implement an automatic, Fail2Ban based permanent IP block at this moment, have some patience.

In general, one can do the following

a) manually add specific bad IPs to the Plesk Firewall ruleset,

b) subscribe to IP Block lists (often not for free and mostly very buggy).

My advice would be to go for option a), since that also reduces the danger of blocking IPs, that should not be blocked.

And Fail2Ban gives you a good insight into which IPs should be blocked with the Plesk Firewall.

For instance, if you set the recidive jail and/or the ssh jail to "2 attempts", you will notice that a lot bad IPs will be blocked temporarily by Fail2Ban.

Those IPs can be used to (manually) create Plesk firewall rules, with an additional advantage: most bad IPs "give up" after some time, so using Fail2Ban to find bad IPs will not keep you busy all the time, the number of permanent IP blocks (in the Plesk Firewall rulesets) can be reasonably reduced.

Hope the above helps...

Kind regards....