High CPU usage for Fail2Ban

[vincheesel]

Hi,

After a restart - the Fail2ban CPU usage is very high!

I've had to disable it because its causing the server to crawl (notice load average is very high)

The data in /var/logs isn't excessive, I've run a logrotate too.
I've tried reinstalling fail2ban too.

Intel Xeon 2640v2 8 core processor (virtualized in hyper-v)
There are about 138 domains (Mainly wordpress CMS)

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=12.04
DISTRIB_CODENAME=precise
DISTRIB_DESCRIPTION="Ubuntu 12.04.5 LTS"

Linux "webserver" 3.2.0-59-generic #90-Ubuntu SMP Tue Jan 7 22:43:51 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

# free mem -g
             total       used       free     shared    buffers     cached
Mem:            15         15          0          0          0         12
-/+ buffers/cache:          2         13
Swap:            1          0          1


top - 00:57:15 up  1:05,  1 user,  load average: 5.61, 6.04, 5.02
Tasks: 222 total,   2 running, 212 sleeping,   0 stopped,   8 zombie
Cpu(s): 14.3%us,  5.9%sy,  3.4%ni, 55.8%id, 20.5%wa,  0.0%hi,  0.1%si,  0.0%st
Mem:  16429136k total, 15854128k used,   575008k free,   396044k buffers
Swap:  2095100k total,        0k used,  2095100k free, 13343500k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
8858 root      20   0 1277m  17m 3040 S  110  0.1  50:27.34 fail2ban-server
19110 sdcpm  20   0  284m  43m 8528 S   15  0.3   0:00.94 php5-cgi
15323 ghhfone  20   0  318m  65m 5272 R   12  0.4   1:13.67 php5-fpm

 

[vincheesel]

/var/log directory

total 873M
-rw-r--r-- 1 root              root   2.2K May  4 23:36 alternatives.log
-rw-r--r-- 1 root              root    611 Jan 12 00:06 alternatives.log.1
-rw-r--r-- 1 root              root    165 Feb 10  2015 alternatives.log.10.gz
-rw-r--r-- 1 root              root    255 Nov  4  2014 alternatives.log.11.gz
-rw-r--r-- 1 root              root    253 Oct  7  2014 alternatives.log.12.gz
-rw-r--r-- 1 root              root    311 Nov 27 23:07 alternatives.log.2.gz
-rw-r--r-- 1 root              root    147 Oct 26  2015 alternatives.log.3.gz
-rw-r--r-- 1 root              root    176 Oct  7  2015 alternatives.log.4.gz
-rw-r--r-- 1 root              root    107 Sep  4  2015 alternatives.log.5.gz
-rw-r--r-- 1 root              root    176 Jul 14  2015 alternatives.log.6.gz
-rw-r--r-- 1 root              root    177 May  9  2015 alternatives.log.7.gz
-rw-r--r-- 1 root              root    176 Apr  7  2015 alternatives.log.8.gz
-rw-r--r-- 1 root              root    290 Mar  9  2015 alternatives.log.9.gz
drwxr-x--- 2 root              adm     12K Feb  7 08:44 apache2
-rw-r----- 1 root              adm     467 Feb 23 15:40 apport.log
-rw-r----- 1 root              adm     866 Dec 14 22:08 apport.log.1
-rw-r----- 1 root              adm     241 Dec 14 03:33 apport.log.2.gz
-rw-r----- 1 root              adm     222 Dec 13 03:41 apport.log.3.gz
-rw-r----- 1 root              adm     239 Dec 12 05:17 apport.log.4.gz
-rw-r----- 1 root              adm     241 Dec 11 05:52 apport.log.5.gz
-rw-r----- 1 root              adm     222 Dec 10 02:53 apport.log.6.gz
-rw-r----- 1 root              adm     223 Dec  9 04:19 apport.log.7.gz
drwxr-xr-x 2 root              root   4.0K Feb  1 08:17 apt
-rw-r--r-- 1 root              root      0 Jun  1  2014 aptitude
-rw-r--r-- 1 root              root    450 May 16  2014 aptitude.1.gz
-rw-r----- 1 syslog            adm     12M May  5 00:55 auth.log
-rw-r----- 1 syslog            adm    825K Feb  7 08:40 auth.log.1
-rw-r----- 1 syslog            adm     60K Jan 31 08:30 auth.log.2.gz
-rw-r----- 1 syslog            adm     50K Jan 24 08:45 auth.log.3.gz
-rw-r----- 1 syslog            adm     41K Jan 17 07:23 auth.log.4.gz
-rw-r----- 1 root              adm      31 Oct  8  2013 boot
-rw-r--r-- 1 root              root   5.3K May  4 23:51 boot.log
-rw-rw---- 1 root              utmp   1.5K Apr 22 14:37 btmp
-rw-rw---- 1 root              utmp    384 Jan 11 11:06 btmp.1
drwxr-xr-x 2 clamav            clamav 4.0K Feb 10 03:00 clamav
drwxr-xr-x 2 root              root   4.0K Apr 21  2012 dist-upgrade
-rw-r----- 1 root              adm     37K May  4 23:51 dmesg
-rw-r----- 1 root              adm     37K May  4 21:24 dmesg.0
-rw-r----- 1 root              adm     11K May  4 18:47 dmesg.1.gz
-rw-r----- 1 root              adm     11K Apr 12 00:48 dmesg.2.gz
-rw-r----- 1 root              adm     11K Mar 15 01:18 dmesg.3.gz
-rw-r----- 1 root              adm     11K Feb 29 23:47 dmesg.4.gz
-rw-r--r-- 1 root              root   341K May  5 00:10 dpkg.log
-rw-r--r-- 1 root              root    85K Jan 21 06:37 dpkg.log.1
-rw-r--r-- 1 root              root   1.9K Apr  7  2015 dpkg.log.10.gz
-rw-r--r-- 1 root              root   3.3K Mar  9  2015 dpkg.log.11.gz
-rw-r--r-- 1 root              root   3.0K Feb 27  2015 dpkg.log.12.gz
-rw-r--r-- 1 root              root   3.2K Dec 28 06:27 dpkg.log.2.gz
-rw-r--r-- 1 root              root   4.2K Nov 27 23:08 dpkg.log.3.gz
-rw-r--r-- 1 root              root    16K Oct 26  2015 dpkg.log.4.gz
-rw-r--r-- 1 root              root   1.8K Sep 24  2015 dpkg.log.5.gz
-rw-r--r-- 1 root              root   2.4K Aug 28  2015 dpkg.log.6.gz
-rw-r--r-- 1 root              root   2.8K Jul 31  2015 dpkg.log.7.gz
-rw-r--r-- 1 root              root   2.0K Jun 19  2015 dpkg.log.8.gz
-rw-r--r-- 1 root              root   3.0K May  9  2015 dpkg.log.9.gz
-rw------- 1 root              root   561K May  5 00:56 fail2ban.log
-rw-r--r-- 1 root              root   318K Apr 28 17:08 faillog
-rw-r--r-- 1 root              root    807 Oct 26  2015 fontconfig.log
drwxr-xr-x 2 root              root   4.0K Oct  8  2013 fsck
drwxr-xr-x 3 root              root   4.0K Oct  8  2013 installer
-rw-r----- 1 syslog            adm    730K May  4 23:56 kern.log
-rw-r----- 1 syslog            adm    106K Feb  7 08:16 kern.log.1
-rw-r----- 1 syslog            adm     28K Jan 30 10:49 kern.log.2.gz
-rw-r----- 1 syslog            adm    2.0K Jan 24 00:40 kern.log.3.gz
-rw-r----- 1 syslog            adm     19K Jan 17 00:24 kern.log.4.gz
drwxr-xr-x 2 landscape         root   4.0K Feb  7 08:44 landscape
-rw-rw-r-- 1 root              utmp   2.9M May  4 23:53 lastlog
-rw-r----- 1 syslog            adm     16M May  5 01:00 mail.err
-rw-r----- 1 syslog            adm    763K Feb  7 08:44 mail.err.1
-rw-r----- 1 syslog            adm     48K Jan 31 08:33 mail.err.2.gz
-rw-r----- 1 syslog            adm     58K Jan 24 08:48 mail.err.3.gz
-rw-r----- 1 syslog            adm     60K Jan 17 07:19 mail.err.4.gz
-rw-r----- 1 syslog            adm    232M May  5 01:00 maillog
-rw-r----- 1 syslog            adm    249M May  5 01:00 mail.log
-rw-r----- 1 syslog            adm     15M Feb  7 08:44 mail.log.1
-rw-r----- 1 syslog            adm    1.9M Jan 31 08:33 mail.log.2.gz
-rw-r----- 1 syslog            adm    5.0M Jan 24 08:48 mail.log.3.gz
-rw-r----- 1 syslog            adm    2.5M Jan 17 07:22 mail.log.4.gz
-rw-r----- 1 root              root    12M Feb 10 06:56 maillog.processed
-rw-r----- 1 root              root   1.2M Feb  9 06:58 maillog.processed.1.gz
-rw-r----- 1 root              root   1.2M Feb  4 07:25 maillog.processed.2.gz
-rw-r----- 1 root              root   1.3M Jan 30 07:17 maillog.processed.3.gz
drwxrwsr-x 2 root              list   4.0K Feb  7 08:44 mailman
-rw-r----- 1 root              root   260K May  5 00:51 modsec_audit.log
-rw-r----- 1 root              root    48M May  4 23:25 modsec_audit.log.1.gz
-rw-r----- 1 root              root   146K Feb 10 08:41 modsec_audit.log.2.gz
-rw-r----- 1 root              root   256K Feb  9 08:10 modsec_audit.log.3.gz
-rw-r----- 1 root              root   203K Feb  8 07:42 modsec_audit.log.4.gz
-rw-r----- 1 root              root   157K Feb  7 08:39 modsec_audit.log.5.gz
-rw-r----- 1 root              root   184K Feb  6 08:04 modsec_audit.log.6.gz
-rw-r----- 1 root              root   222K Feb  5 07:43 modsec_audit.log.7.gz
drwxr-s--- 2 mysql             adm    4.0K Feb 10 08:42 mysql
-rw-r----- 1 mysql             adm       0 May  4 23:35 mysql.err
-rw-r----- 1 mysql             adm       0 May  4 23:35 mysql.log
-rw-r----- 1 mysql             adm      20 Feb  9 08:10 mysql.log.1.gz
-rw-r----- 1 mysql             adm      20 Feb  8 07:55 mysql.log.2.gz
-rw-r----- 1 mysql             adm      20 Feb  7 08:44 mysql.log.3.gz
-rw-r----- 1 mysql             adm      20 Feb  6 08:05 mysql.log.4.gz
-rw-r----- 1 mysql             adm      20 Feb  5 07:44 mysql.log.5.gz
-rw-r----- 1 mysql             adm      20 Feb  4 08:13 mysql.log.6.gz
-rw-r----- 1 mysql             adm      20 Feb  3 08:37 mysql.log.7.gz
drwxr-xr-x 2 root              root   4.0K Oct  8  2013 news
drwxr-xr-x 2 root              root   4.0K Feb 10 08:42 nginx
-rw------- 1 root              root   3.2M May  4 23:51 php5-fpm.log
drwxr-x--- 5 psaadm            root   4.0K Feb 10 06:33 plesk
drwxr-xr-x 2 root              root   4.0K Oct 21  2015 plesk-php54-fpm
drwxr-xr-x 2 root              root   4.0K Oct 21  2015 plesk-php55-fpm
drwxr-xr-x 2 root              root   4.0K Feb 10 11:54 plesk-php56-fpm
drwxr-xr-x 2 root              root   4.0K May  4 23:51 plesk-php70-fpm
drwxr-x--- 2 roundcube_sysuser root   4.0K Feb  4 07:12 plesk-roundcube
drwxr-xr-x 2 horde_sysuser     root   4.0K Nov  1  2013 psa-horde
drwxr-x--- 2 root              adm    4.0K Sep 24  2013 samba
drwxr-x--- 2 root              root   4.0K Feb  7 08:44 sw-cp-server
-rw-r----- 1 syslog            adm    254M May  5 01:00 syslog
-rw-r----- 1 syslog            adm     13M Feb 10 08:42 syslog.1
-rw-r----- 1 syslog            adm    361K Feb  9 08:10 syslog.2.gz
-rw-r----- 1 syslog            adm    364K Feb  8 07:55 syslog.3.gz
-rw-r----- 1 syslog            adm    207K Feb  7 08:44 syslog.4.gz
-rw-r----- 1 syslog            adm    299K Feb  6 08:05 syslog.5.gz
-rw-r----- 1 syslog            adm    319K Feb  5 07:44 syslog.6.gz
-rw-r----- 1 syslog            adm    305K Feb  4 08:13 syslog.7.gz
drwxr-xr-x 2 root              root   4.0K Dec 17  2011 sysstat
-rw-r--r-- 1 root              root   2.7K Nov 28 00:01 trueimage-setup.log
-rw-r--r-- 1 root              root   170K May  4 23:51 udev
-rw-r----- 1 syslog            adm       0 Oct  8  2013 ufw.log
drwxr-xr-x 2 root              root   4.0K Nov 15  2012 unattended-upgrades
drwxr-xr-x 2 root              root   4.0K Feb 29 23:18 upstart
-rw-rw-r-- 1 root              utmp   1.6M May  4 23:53 wtmp
-rw-rw-r-- 1 root              utmp   379K Jan 31 03:56 wtmp.1

 

[IgorG]

I think this problem is addressed in the following KB - https://kb.plesk.com/en/122407

 

[vincheesel]

Thanks Igor,

I tried the resolution part - unfortunately it failed - so I need to make smaller jails

I'm not sure if I understand the part for larger domains on the bottom - We don't have reseller accounts and just the 1 admin account for all domains, which is just "admin"
Could you please elaborate on how to proceed here?

1. Get the admin email:
   
   admin_email=`mysql -Ns -uadmin -p\`cat /etc/psa/.psa.shadow\` psa -Ne"select email from clients where login='admin'"`
   
   
2. Set plesk-apache jails:
   
   for i in a b c d e f g h i j k l m n o p q r s t u v w x y z 1 2 3 4 5 6 7 8 9 0;do find /var/www/vhosts/system/$i*/logs/error_log 2>/dev/null 1>/dev/null; found=`echo $?`;if [ $found == "0" ];then echo "[[\"usedns\",\"no\"],[\"logpath\",\"\\/var\\/www\\/vhosts\\/system\\/$i*\\/logs\\/error_log\"],[\"enabled\",\"true\"],[\"filter\",\"apache-auth\"],[\"maxretry\",\"6\"],[\"__source__\",\"jail.d\\/plesk.conf\"],[\"action\",\"iptables-multiport[name=apache, port=\\\"http,https,7080,7081\\\"]\"],[\"ignoreip\",\"127.0.0.1\/8\"],[\"bantime\",\"600\"],[\"destemail\",\"$admin_email\"],[\"findtime\",\"600\"],[\"backend\",\"auto\"]]"|/usr/local/psa/admin/bin/f2bmng --set-jail plesk-apache-$i ;fi;done

 

[vincheesel]

bumpp