Resolved Fail2ban findFailure errors lead to disk full

Pascal_Netenvie · Apr 26, 2016

Hi,
I'm fighting a strange problem with Fail2ban.
There is an error happening 1000 times per second and filling log file very fast (1000 lines per second).
After some hours log is 45GB size and disk is full so server goes down.

This is the error :

2016-04-26 19:29:40,999 fail2ban.filter [13939]: ERROR findFailure failed to parse timeText: Apr 25 06:42:50 1008

What cause that ?
How to solve it ?

This is server specs :
Plesk v12.5.30_build1205150826.19 os_Debian 8.0
OS Debian 8.4

I tried to uninstall then install Plesk component Fail2ban but it changed nothing.

IgorG · Apr 26, 2016

Something wrong in fail2ban configs. You can just backup it with

#mv /etc/fail2ban /etc/fail2ban.old

and after that reinstall fail2ban with autoinstaller.
Also following article may be useful - https://github.com/fail2ban/fail2ban/issues/1278

Pascal_Netenvie · Apr 28, 2016

Hi,
As you said i moved config folder then i removed and installed Fail2ban with autoinstaller.

As soon as i activate ProFTPD or SSH jail the log start to grow.

This is last lines in log before bug start with FTPD :

Code:

2016-04-28 18:01:34,280 fail2ban.transmitter    [15115]: WARNING Command ['stop', 'plesk-proftpd'] has failed. Received UnknownJailException('plesk-proftpd',)
2016-04-28 18:01:34,282 fail2ban.server         [15115]: INFO    Changed logging target to /var/log/fail2ban.log for Fail2ban v0.9.2
2016-04-28 18:01:34,284 fail2ban.jail           [15115]: INFO    Creating new jail 'plesk-proftpd'
2016-04-28 18:01:34,284 fail2ban.jail           [15115]: INFO    Jail 'plesk-proftpd' uses pyinotify
2016-04-28 18:01:34,285 fail2ban.filter         [15115]: INFO    Set jail log file encoding to UTF-8
2016-04-28 18:01:34,290 fail2ban.jail           [15115]: INFO    Initiated 'pyinotify' backend
2016-04-28 18:01:34,342 fail2ban.filter         [15115]: INFO    Added logfile = /var/log/auth.log
2016-04-28 18:01:34,365 fail2ban.filter         [15115]: INFO    Set maxRetry = 5
2016-04-28 18:01:34,367 fail2ban.filter         [15115]: INFO    Set findtime = 600
2016-04-28 18:01:34,367 fail2ban.actions        [15115]: INFO    Set banTime = 600
2016-04-28 18:01:34,389 fail2ban.jail           [15115]: INFO    Jail 'plesk-proftpd' started
2016-04-28 18:01:38,819 fail2ban.filter         [15115]: ERROR   findFailure failed to parse timeText: Apr 25 06:25:16 1008
2016-04-28 18:01:38,820 fail2ban.filter         [15115]: ERROR   findFailure failed to parse timeText: Apr 25 06:25:16 1008
2016-04-28 18:01:38,821 fail2ban.filter         [15115]: ERROR   findFailure failed to parse timeText: Apr 25 06:25:16 1008
2016-04-28 18:01:38,822 fail2ban.filter         [15115]: ERROR   findFailure failed to parse timeText: Apr 25 06:25:16 1008
2016-04-28 18:01:38,822 fail2ban.filter         [15115]: ERROR   findFailure failed to parse timeText: Apr 25 06:25:16 1008
2016-04-28 18:01:38,822 fail2ban.filter         [15115]: ERROR   findFailure failed to parse timeText: Apr 25 06:25:17 1008
2016-04-28 18:01:38,823 fail2ban.filter         [15115]: ERROR   findFailure failed to parse timeText: Apr 25 06:25:18 1008
2016-04-28 18:01:38,823 fail2ban.filter         [15115]: ERROR   findFailure failed to parse timeText: Apr 25 06:25:18 1008
2016-04-28 18:01:38,823 fail2ban.filter         [15115]: ERROR   findFailure failed to parse timeText: Apr 25 06:25:20 1008
2016-04-28 18:01:38,824 fail2ban.filter         [15115]: ERROR   findFailure failed to parse timeText: Apr 25 06:25:20 1008
2016-04-28 18:01:38,824 fail2ban.filter         [15115]: ERROR   findFailure failed to parse timeText: Apr 25 06:25:20 1008
2016-04-28 18:01:38,824 fail2ban.filter         [15115]: ERROR   findFailure failed to parse timeText: Apr 25 06:25:21 1008
2016-04-28 18:01:38,824 fail2ban.filter         [15115]: ERROR   findFailure failed to parse timeText: Apr 25 06:25:21 1008

Same with SSH :

Code:

2016-04-28 18:09:37,151 fail2ban.jail           [15115]: INFO    Creating new jail 'ssh'
2016-04-28 18:09:37,151 fail2ban.jail           [15115]: INFO    Jail 'ssh' uses pyinotify
2016-04-28 18:09:37,153 fail2ban.filter         [15115]: INFO    Set jail log file encoding to UTF-8
2016-04-28 18:09:37,155 fail2ban.filter         [15115]: INFO    Log rotation detected for /var/log/fail2ban.log
2016-04-28 18:09:37,160 fail2ban.jail           [15115]: INFO    Initiated 'pyinotify' backend
2016-04-28 18:09:37,194 fail2ban.filter         [15115]: INFO    Added logfile = /var/log/auth.log
2016-04-28 18:09:37,207 fail2ban.filter         [15115]: INFO    Set maxRetry = 5
2016-04-28 18:09:37,208 fail2ban.filter         [15115]: INFO    Set findtime = 600
2016-04-28 18:09:37,209 fail2ban.actions        [15115]: INFO    Set banTime = 600
2016-04-28 18:09:37,210 fail2ban.filter         [15115]: INFO    Set maxlines = 10
2016-04-28 18:09:37,234 fail2ban.server         [15115]: INFO    Jail ssh is not a JournalFilter instance
2016-04-28 18:09:37,245 fail2ban.jail           [15115]: INFO    Jail 'ssh' started
2016-04-28 18:09:37,505 fail2ban.filter         [15115]: ERROR   findFailure failed to parse timeText: Apr 25 06:25:16 1008
2016-04-28 18:09:37,506 fail2ban.filter         [15115]: WARNING Found a match for u'Apr 25 06:25:16 100852hpv124114 sshd[24194]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=183.3.202.199  user=root' but no valid date/time found for u'Apr 25 06:25:16 1008'. Please try setting a custom date pattern (see man page jail.conf(5)). If format is complex, please file a detailed issue on https://github.com/fail2ban/fail2ban/issues in order to get support for this format.
2016-04-28 18:09:37,506 fail2ban.filter         [15115]: ERROR   findFailure failed to parse timeText: Apr 25 06:25:16 1008
2016-04-28 18:09:37,506 fail2ban.filter         [15115]: WARNING Found a match for u'Apr 25 06:25:16 100852hpv124114 sshd[24337]: Invalid user thierry from 93.63.230.189' but no valid date/time found for u'Apr 25 06:25:16 1008'. Please try setting a custom date pattern (see man page jail.conf(5)). If format is complex, please file a detailed issue on https://github.com/fail2ban/fail2ban/issues in order to get support for this format.
2016-04-28 18:09:37,507 fail2ban.filter         [15115]: WARNING Found a match for u'Apr 25 06:25:16 100852hpv124114 sshd[24194]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=183.3.202.199  user=root' but no valid date/time found for u'Apr 25 06:25:16 1008'. Please try setting a custom date pattern (see man page jail.conf(5)). If format is complex, please file a detailed issue on https://github.com/fail2ban/fail2ban/issues in order to get support for this format.
2016-04-28 18:09:37,507 fail2ban.filter         [15115]: ERROR   findFailure failed to parse timeText: Apr 25 06:25:16 1008
2016-04-28 18:09:37,510 fail2ban.filter         [15115]: WARNING Found a match for u'Apr 25 06:25:16 100852hpv124114 sshd[24337]: Invalid user thierry from 93.63.230.189' but no valid date/time found for u'Apr 25 06:25:16 1008'. Please try setting a custom date pattern (see man page jail.conf(5)). If format is complex, please file a detailed issue on https://github.com/fail2ban/fail2ban/issues in order to get support for this format.
2016-04-28 18:09:37,510 fail2ban.filter         [15115]: WARNING Found a match for u'Apr 25 06:25:16 100852hpv124114 sshd[24194]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=183.3.202.199  user=root' but no valid date/time found for u'Apr 25 06:25:16 1008'. Please try setting a custom date pattern (see man page jail.conf(5)). If format is complex, please file a detailed issue on https://github.com/fail2ban/fail2ban/issues in order to get support for this format.

IgorG · Apr 28, 2016

What is output of date command?

Pascal_Netenvie · Apr 29, 2016

It is present time :
vendredi 29 avril 2016, 09:21:40 (UTC+0200)

Pascal_Netenvie · May 4, 2016

Still can't enable SSH and FTPD jails on fail2ban ...

IgorG · May 4, 2016

Try to set en_US.UTF-8 locale for console.
Output should be in English like

# date
Wed May 4 14:36:50 NOVT 2016

Pascal_Netenvie · May 4, 2016

IgorG said:
Try to set en_US.UTF-8 locale for console

How to set that ?

IgorG · May 4, 2016

Pascal_Netenvie said:
How to set that ?

https://talk.plesk.com/threads/warning-current-locale-is-unusable-using-c-instead.331221

Pascal_Netenvie · May 24, 2016

This is output for locale command :

Code:

# locale
LANG=fr_FR.UTF-8
LANGUAGE=
LC_CTYPE="fr_FR.UTF-8"
LC_NUMERIC="fr_FR.UTF-8"
LC_TIME="fr_FR.UTF-8"
LC_COLLATE="fr_FR.UTF-8"
LC_MONETARY="fr_FR.UTF-8"
LC_MESSAGES="fr_FR.UTF-8"
LC_PAPER="fr_FR.UTF-8"
LC_NAME="fr_FR.UTF-8"
LC_ADDRESS="fr_FR.UTF-8"
LC_TELEPHONE="fr_FR.UTF-8"
LC_MEASUREMENT="fr_FR.UTF-8"
LC_IDENTIFICATION="fr_FR.UTF-8"
LC_ALL=

All seems utf8.
Why should i change from fr_FR to en_US as server use french language ?
And server date and time are correct ...

Pascal_Netenvie · May 24, 2016

Funny,
i just have now the problem on another server installed 4 days ago ...

Pascal_Netenvie · May 24, 2016

Date and locale for this new server :

Code:

# date
mardi 24 mai 2016, 17:23:24 (UTC+0200)

# locale
LANG=fr_FR.UTF-8
LANGUAGE=
LC_CTYPE="fr_FR.UTF-8"
LC_NUMERIC="fr_FR.UTF-8"
LC_TIME="fr_FR.UTF-8"
LC_COLLATE="fr_FR.UTF-8"
LC_MONETARY="fr_FR.UTF-8"
LC_MESSAGES="fr_FR.UTF-8"
LC_PAPER="fr_FR.UTF-8"
LC_NAME="fr_FR.UTF-8"
LC_ADDRESS="fr_FR.UTF-8"
LC_TELEPHONE="fr_FR.UTF-8"
LC_MEASUREMENT="fr_FR.UTF-8"
LC_IDENTIFICATION="fr_FR.UTF-8"
LC_ALL=

Pascal_Netenvie · May 24, 2016

Ok so i tried this :

Code:

export LC_ALL="en_US"

Then i have deleted fail2ban config folder
then i removed and installed Fail2ban with autoinstaller.

Problem still the same ...

Is the problem come from locale settings ?

Pascal_Netenvie · May 24, 2016

It didn't fixed anything ...
Always this error :

Code:

2016-05-24 17:54:57,407 fail2ban.filter         [25812]: ERROR   findFailure failed to parse timeText: May 23 08:47:11 1014
2016-05-24 17:54:57,407 fail2ban.filter         [25812]: WARNING Found a match for u'May 23 08:47:00 101496hpv124170 sshd[46053]: Failed password for root from 183.3.202.199 port 14423 ssh2' but no valid date/time found for u'May 23 08:47:11 1014'. Please try setting a custom date pattern (see man page jail.conf(5)). If format is complex, please file a detailed issue on https://github.com/fail2ban/fail2ban/issues in order to get support for this format.

Pascal_Netenvie · May 24, 2016

As i look in log i think fail2ban try to parse date with a pattern beginning by year on 4 digits.
And here there is no year so it take 4 more character at end and get 4 first digit of server ID ... leading to a date it can't parse.
An idea to fix this ?

Pascal_Netenvie · May 24, 2016

Ok so finally it seems i found how to fix this problem.
I have to add a date pattern in jail filter config.
For example for recidive jail i edited this file :
/etc/fail2ban/filter.d/recidive.conf

and in [Init] section i added this to match my date format :
datepattern = %%b %%d %%H:%%M:%%S

and now it run ok.
Source that helped : https://github.com/fail2ban/fail2ban/issues/1278

Do i have to do this for all jails ??
And why suddenly, on last servers, we have this problem ?

Pascal_Netenvie · May 24, 2016

Humm
Perhaps we can just add date pattern to this file :
/etc/fail2ban/filter.d/common.conf

Can you validate it Igor ?

Pascal_Netenvie · May 24, 2016

I've tested and it seems ok now

But i'm curious to know why we have this problem now !

Dragonz · Jul 17, 2016

Hi, I've the same problem.

Code:

#date
Mon Jul 18 08:49:20 CEST 2016

#locale
LANG=
LANGUAGE=
LC_CTYPE="POSIX"
LC_NUMERIC="POSIX"
LC_TIME="POSIX"
LC_COLLATE="POSIX"
LC_MONETARY="POSIX"
LC_MESSAGES="POSIX"
LC_PAPER="POSIX"
LC_NAME="POSIX"
LC_ADDRESS="POSIX"
LC_TELEPHONE="POSIX"
LC_MEASUREMENT="POSIX"
LC_IDENTIFICATION="POSIX"
LC_ALL=

Resolved Fail2ban findFailure errors lead to disk full

Pascal_Netenvie

Regular Pleskian

IgorG

Plesk addicted!

Pascal_Netenvie

Regular Pleskian

IgorG

Plesk addicted!

Pascal_Netenvie

Regular Pleskian

Pascal_Netenvie

Regular Pleskian

IgorG

Plesk addicted!

Pascal_Netenvie

Regular Pleskian

IgorG

Plesk addicted!

Pascal_Netenvie

Regular Pleskian

Pascal_Netenvie

Regular Pleskian

Pascal_Netenvie

Regular Pleskian

Pascal_Netenvie

Regular Pleskian

Pascal_Netenvie

Regular Pleskian

Pascal_Netenvie

Regular Pleskian

Pascal_Netenvie

Regular Pleskian

Pascal_Netenvie

Regular Pleskian

Pascal_Netenvie

Regular Pleskian

Dragonz

New Pleskian