• If you are still using CentOS 7.9, it's time to convert to Alma 8 with the free centos2alma tool by Plesk or Plesk Migrator. Please let us know your experiences or concerns in this thread:
    CentOS2Alma discussion

Resolved Fail2ban findFailure errors lead to disk full

Pascal_Netenvie

Regular Pleskian
Hi,
I'm fighting a strange problem with Fail2ban.
There is an error happening 1000 times per second and filling log file very fast (1000 lines per second).
After some hours log is 45GB size and disk is full so server goes down.

This is the error :

2016-04-26 19:29:40,999 fail2ban.filter [13939]: ERROR findFailure failed to parse timeText: Apr 25 06:42:50 1008

What cause that ?
How to solve it ?

This is server specs :
Plesk v12.5.30_build1205150826.19 os_Debian 8.0
OS Debian 8.4

I tried to uninstall then install Plesk component Fail2ban but it changed nothing.
 
Hi,
As you said i moved config folder then i removed and installed Fail2ban with autoinstaller.

As soon as i activate ProFTPD or SSH jail the log start to grow.

This is last lines in log before bug start with FTPD :

Code:
2016-04-28 18:01:34,280 fail2ban.transmitter    [15115]: WARNING Command ['stop', 'plesk-proftpd'] has failed. Received UnknownJailException('plesk-proftpd',)
2016-04-28 18:01:34,282 fail2ban.server         [15115]: INFO    Changed logging target to /var/log/fail2ban.log for Fail2ban v0.9.2
2016-04-28 18:01:34,284 fail2ban.jail           [15115]: INFO    Creating new jail 'plesk-proftpd'
2016-04-28 18:01:34,284 fail2ban.jail           [15115]: INFO    Jail 'plesk-proftpd' uses pyinotify
2016-04-28 18:01:34,285 fail2ban.filter         [15115]: INFO    Set jail log file encoding to UTF-8
2016-04-28 18:01:34,290 fail2ban.jail           [15115]: INFO    Initiated 'pyinotify' backend
2016-04-28 18:01:34,342 fail2ban.filter         [15115]: INFO    Added logfile = /var/log/auth.log
2016-04-28 18:01:34,365 fail2ban.filter         [15115]: INFO    Set maxRetry = 5
2016-04-28 18:01:34,367 fail2ban.filter         [15115]: INFO    Set findtime = 600
2016-04-28 18:01:34,367 fail2ban.actions        [15115]: INFO    Set banTime = 600
2016-04-28 18:01:34,389 fail2ban.jail           [15115]: INFO    Jail 'plesk-proftpd' started
2016-04-28 18:01:38,819 fail2ban.filter         [15115]: ERROR   findFailure failed to parse timeText: Apr 25 06:25:16 1008
2016-04-28 18:01:38,820 fail2ban.filter         [15115]: ERROR   findFailure failed to parse timeText: Apr 25 06:25:16 1008
2016-04-28 18:01:38,821 fail2ban.filter         [15115]: ERROR   findFailure failed to parse timeText: Apr 25 06:25:16 1008
2016-04-28 18:01:38,822 fail2ban.filter         [15115]: ERROR   findFailure failed to parse timeText: Apr 25 06:25:16 1008
2016-04-28 18:01:38,822 fail2ban.filter         [15115]: ERROR   findFailure failed to parse timeText: Apr 25 06:25:16 1008
2016-04-28 18:01:38,822 fail2ban.filter         [15115]: ERROR   findFailure failed to parse timeText: Apr 25 06:25:17 1008
2016-04-28 18:01:38,823 fail2ban.filter         [15115]: ERROR   findFailure failed to parse timeText: Apr 25 06:25:18 1008
2016-04-28 18:01:38,823 fail2ban.filter         [15115]: ERROR   findFailure failed to parse timeText: Apr 25 06:25:18 1008
2016-04-28 18:01:38,823 fail2ban.filter         [15115]: ERROR   findFailure failed to parse timeText: Apr 25 06:25:20 1008
2016-04-28 18:01:38,824 fail2ban.filter         [15115]: ERROR   findFailure failed to parse timeText: Apr 25 06:25:20 1008
2016-04-28 18:01:38,824 fail2ban.filter         [15115]: ERROR   findFailure failed to parse timeText: Apr 25 06:25:20 1008
2016-04-28 18:01:38,824 fail2ban.filter         [15115]: ERROR   findFailure failed to parse timeText: Apr 25 06:25:21 1008
2016-04-28 18:01:38,824 fail2ban.filter         [15115]: ERROR   findFailure failed to parse timeText: Apr 25 06:25:21 1008

Same with SSH :

Code:
2016-04-28 18:09:37,151 fail2ban.jail           [15115]: INFO    Creating new jail 'ssh'
2016-04-28 18:09:37,151 fail2ban.jail           [15115]: INFO    Jail 'ssh' uses pyinotify
2016-04-28 18:09:37,153 fail2ban.filter         [15115]: INFO    Set jail log file encoding to UTF-8
2016-04-28 18:09:37,155 fail2ban.filter         [15115]: INFO    Log rotation detected for /var/log/fail2ban.log
2016-04-28 18:09:37,160 fail2ban.jail           [15115]: INFO    Initiated 'pyinotify' backend
2016-04-28 18:09:37,194 fail2ban.filter         [15115]: INFO    Added logfile = /var/log/auth.log
2016-04-28 18:09:37,207 fail2ban.filter         [15115]: INFO    Set maxRetry = 5
2016-04-28 18:09:37,208 fail2ban.filter         [15115]: INFO    Set findtime = 600
2016-04-28 18:09:37,209 fail2ban.actions        [15115]: INFO    Set banTime = 600
2016-04-28 18:09:37,210 fail2ban.filter         [15115]: INFO    Set maxlines = 10
2016-04-28 18:09:37,234 fail2ban.server         [15115]: INFO    Jail ssh is not a JournalFilter instance
2016-04-28 18:09:37,245 fail2ban.jail           [15115]: INFO    Jail 'ssh' started
2016-04-28 18:09:37,505 fail2ban.filter         [15115]: ERROR   findFailure failed to parse timeText: Apr 25 06:25:16 1008
2016-04-28 18:09:37,506 fail2ban.filter         [15115]: WARNING Found a match for u'Apr 25 06:25:16 100852hpv124114 sshd[24194]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=183.3.202.199  user=root' but no valid date/time found for u'Apr 25 06:25:16 1008'. Please try setting a custom date pattern (see man page jail.conf(5)). If format is complex, please file a detailed issue on https://github.com/fail2ban/fail2ban/issues in order to get support for this format.
2016-04-28 18:09:37,506 fail2ban.filter         [15115]: ERROR   findFailure failed to parse timeText: Apr 25 06:25:16 1008
2016-04-28 18:09:37,506 fail2ban.filter         [15115]: WARNING Found a match for u'Apr 25 06:25:16 100852hpv124114 sshd[24337]: Invalid user thierry from 93.63.230.189' but no valid date/time found for u'Apr 25 06:25:16 1008'. Please try setting a custom date pattern (see man page jail.conf(5)). If format is complex, please file a detailed issue on https://github.com/fail2ban/fail2ban/issues in order to get support for this format.
2016-04-28 18:09:37,507 fail2ban.filter         [15115]: WARNING Found a match for u'Apr 25 06:25:16 100852hpv124114 sshd[24194]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=183.3.202.199  user=root' but no valid date/time found for u'Apr 25 06:25:16 1008'. Please try setting a custom date pattern (see man page jail.conf(5)). If format is complex, please file a detailed issue on https://github.com/fail2ban/fail2ban/issues in order to get support for this format.
2016-04-28 18:09:37,507 fail2ban.filter         [15115]: ERROR   findFailure failed to parse timeText: Apr 25 06:25:16 1008
2016-04-28 18:09:37,510 fail2ban.filter         [15115]: WARNING Found a match for u'Apr 25 06:25:16 100852hpv124114 sshd[24337]: Invalid user thierry from 93.63.230.189' but no valid date/time found for u'Apr 25 06:25:16 1008'. Please try setting a custom date pattern (see man page jail.conf(5)). If format is complex, please file a detailed issue on https://github.com/fail2ban/fail2ban/issues in order to get support for this format.
2016-04-28 18:09:37,510 fail2ban.filter         [15115]: WARNING Found a match for u'Apr 25 06:25:16 100852hpv124114 sshd[24194]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=183.3.202.199  user=root' but no valid date/time found for u'Apr 25 06:25:16 1008'. Please try setting a custom date pattern (see man page jail.conf(5)). If format is complex, please file a detailed issue on https://github.com/fail2ban/fail2ban/issues in order to get support for this format.
 
Try to set en_US.UTF-8 locale for console.
Output should be in English like

# date
Wed May 4 14:36:50 NOVT 2016
 
This is output for locale command :
Code:
# locale
LANG=fr_FR.UTF-8
LANGUAGE=
LC_CTYPE="fr_FR.UTF-8"
LC_NUMERIC="fr_FR.UTF-8"
LC_TIME="fr_FR.UTF-8"
LC_COLLATE="fr_FR.UTF-8"
LC_MONETARY="fr_FR.UTF-8"
LC_MESSAGES="fr_FR.UTF-8"
LC_PAPER="fr_FR.UTF-8"
LC_NAME="fr_FR.UTF-8"
LC_ADDRESS="fr_FR.UTF-8"
LC_TELEPHONE="fr_FR.UTF-8"
LC_MEASUREMENT="fr_FR.UTF-8"
LC_IDENTIFICATION="fr_FR.UTF-8"
LC_ALL=

All seems utf8.
Why should i change from fr_FR to en_US as server use french language ?
And server date and time are correct ...
 
Date and locale for this new server :
Code:
# date
mardi 24 mai 2016, 17:23:24 (UTC+0200)

# locale
LANG=fr_FR.UTF-8
LANGUAGE=
LC_CTYPE="fr_FR.UTF-8"
LC_NUMERIC="fr_FR.UTF-8"
LC_TIME="fr_FR.UTF-8"
LC_COLLATE="fr_FR.UTF-8"
LC_MONETARY="fr_FR.UTF-8"
LC_MESSAGES="fr_FR.UTF-8"
LC_PAPER="fr_FR.UTF-8"
LC_NAME="fr_FR.UTF-8"
LC_ADDRESS="fr_FR.UTF-8"
LC_TELEPHONE="fr_FR.UTF-8"
LC_MEASUREMENT="fr_FR.UTF-8"
LC_IDENTIFICATION="fr_FR.UTF-8"
LC_ALL=
 
Ok so i tried this :
Code:
export LC_ALL="en_US"

Then i have deleted fail2ban config folder
then i removed and installed Fail2ban with autoinstaller.

Problem still the same ...

Is the problem come from locale settings ?
 
Last edited:
It didn't fixed anything ...
Always this error :
Code:
2016-05-24 17:54:57,407 fail2ban.filter         [25812]: ERROR   findFailure failed to parse timeText: May 23 08:47:11 1014
2016-05-24 17:54:57,407 fail2ban.filter         [25812]: WARNING Found a match for u'May 23 08:47:00 101496hpv124170 sshd[46053]: Failed password for root from 183.3.202.199 port 14423 ssh2' but no valid date/time found for u'May 23 08:47:11 1014'. Please try setting a custom date pattern (see man page jail.conf(5)). If format is complex, please file a detailed issue on https://github.com/fail2ban/fail2ban/issues in order to get support for this format.
 
Last edited:
As i look in log i think fail2ban try to parse date with a pattern beginning by year on 4 digits.
And here there is no year so it take 4 more character at end and get 4 first digit of server ID ... leading to a date it can't parse.
An idea to fix this ?
 
Last edited:
Ok so finally it seems i found how to fix this problem.
I have to add a date pattern in jail filter config.
For example for recidive jail i edited this file :
/etc/fail2ban/filter.d/recidive.conf

and in [Init] section i added this to match my date format :
datepattern = %%b %%d %%H:%%M:%%S

and now it run ok.
Source that helped : https://github.com/fail2ban/fail2ban/issues/1278

Do i have to do this for all jails ??
And why suddenly, on last servers, we have this problem ?
 
Hi, I've the same problem.

Code:
#date
Mon Jul 18 08:49:20 CEST 2016

#locale
LANG=
LANGUAGE=
LC_CTYPE="POSIX"
LC_NUMERIC="POSIX"
LC_TIME="POSIX"
LC_COLLATE="POSIX"
LC_MONETARY="POSIX"
LC_MESSAGES="POSIX"
LC_PAPER="POSIX"
LC_NAME="POSIX"
LC_ADDRESS="POSIX"
LC_TELEPHONE="POSIX"
LC_MEASUREMENT="POSIX"
LC_IDENTIFICATION="POSIX"
LC_ALL=
 
Back
Top