Forwarded to devs Mod_Security (with Comodo ruleset) IP persistent storage not being cleaned/rotated

[Alban Staehli]

TITLE:

Mod_Security (with Comodo ruleset) IP persistent storage not being cleaned/rotated​

PRODUCT, VERSION, OPERATING SYSTEM, ARCHITECTURE:

Plesk Onyx, 17.0.17 Update #28 & 17.5.3 Update #22, Centos 6.9 / Centos 7.4, x64​

PROBLEM DESCRIPTION:

When using modsec & the comodo ruleset, /var/cache/modsecurity/ip.pag is not being cleaned/rotated. Therefore, after a couple of weeks, it easily reaches multiple GO in size and slow down apache.

More details here:
Issue - Mod_Security IP persistent storage massive

Thanks to @websavers for reporting it.​

STEPS TO REPRODUCE:

1) Enable modsec via Plesk
2) Setup modsec to use the Comodo ruleset
3) Heavily use your web server over a couple of days / weeks
4) Check the size of file /var/cache/modsecurity/ip.pag => it's always growing and is never being rotated/cleaned​

ACTUAL RESULT:

/var/cache/modsecurity/ip.pag is always growing and is never being rotated/cleaned - therefore it slows down apache when reaching a concerning size​

EXPECTED RESULT:

/var/cache/modsecurity/ip.pag should be regularly cleaned/rotated, even with Comodo ruleset.

Workaround:
cron job run daily like this: cat /dev/null > /var/cache/modsecurity/ip.dir && cat /dev/null > /var/cache/modsecurity/ip.pag​

ANY ADDITIONAL INFORMATION:

Seems to be a known modsec bug:
IP persistence storage seems to not clean up · Issue #574 · SpiderLabs/ModSecurity · GitHub​

YOUR EXPECTATIONS FROM PLESK SERVICE TEAM:

Confirm bug​

 

[IgorG]

According to developers this is subject for investigation by modsecurity support. Please contact them.
Thanks.

 

[Alban Staehli]

According to developers this is subject for investigation by modsecurity support. Please contact them.
Thanks.

Thx for this helpful feedback. Nevertheless, in Web Application Firewall (ModSecurity) it is mentionned that "Plesk will install its own ModSecurity package". Therefore, can you please indicate which version of modsec is being shipped with latest Plesk Onyx 17.5.3 Update #24?

 

[UFHH01]

Hi Alban Staehli,

Therefore, can you please indicate which version of modsec is being shipped with latest Plesk Onyx 17.5.3 Update #24?

Pls. have a look at your current ModSecurity VERSION on your server with the example command:

cat /etc/asl/VERSION

 

[Alban Staehli]

Thx - just to mention it has got even worst now with COMODO. Had to switch to Atomic.
See mod_security2: Failed to write to DBM file "/var/cache/modsecurity/ip": Invalid argument

BTW - in the link above, there's a link to GitHub - SpiderLabs/modsec-sdbm-util: Utility to manipulate SDBM files used by ModSecurity. With that utility it is possible to _shrink_ SDBM databases. It is also possible to list the SDBM contents with filters such as: expired or invalid items only. - might be cleaner than emptying the ip.dag file manually.

 

[websavers]

According to developers this is subject for investigation by modsecurity support. Please contact them.
Thanks.

Yes, and the modsecurity developers have already fixed this with a patch applied to modsecurity 2.9.2 as shown on GitHub here.

Since the version of mod security we have installed on Plesk Onyx Version 17.5.3 Update #28 is packaged by Plesk devs with modsec version 2.9.0, it would be up to you guys to update your packages with either a newer version of modsec or backport the patch to the 2.9.0 packages of modsec you're using now.

[root]# rpm -qi mod_security-2.9.0-centos6.17031414.x86_64
Name        : mod_security                 Relocations: (not relocatable)
Version     : 2.9.0                             Vendor: Plesk
Release     : centos6.17031414              Build Date: Tue 14 Mar 2017 03:53:33 AM EDT
Install Date: Sun 05 Nov 2017 08:29:18 PM EST      Build Host: bcos6x64.plesk.ru
Group       : System Environment/Daemons    Source RPM: mod_security-2.9.0-centos6.17031414.src.rpm
Size        : 1051297                          License: Apache License 2.0
Signature   : DSA/SHA1, Tue 14 Mar 2017 06:25:32 AM EDT, Key ID bd11a6aa914bdf7e
Packager    : Plesk <[email protected]>
URL         : http://www.modsecurity.org/

# cat /etc/asl/VERSION
ASL_VERSION=5.0-1426
APPINV_VERSION=0
CLAMAV_VERSION=0
GEOMAP_VERSION=0
GRSEC_VERSION=0
KERNEL_VERSION=0
MODSEC_VERSION=201704151516
OSSEC_VERSION=0
WAF_DELAYED_VERSION=0

 

[websavers]

Note that I've completed *all* of the steps found here, to supposedly fix this problem. This includes:

1. Updating the ruleset and confirming its updated version
2. Turning off the web app firewall, clearing the ip.pag file, then re-enabling the firewall
3. Disabling the Bruteforce ruleset
4. Setting SecCollectionTimeout to just 30 seconds

Yet our CentOS 6 machines all still end up with 8GB+ ip.pag files over time. This problem appears to *require* that mod security fix. Why has the binary still not been patched?

 

[tmuka]

I'm experiencing this issue with plesk-modsecurity files in /var/cache/modsecurity getting so large that modsec fails to delete from the cache. When that happens, /var/log/modsec_audit.log gets thousands of errors like this one

--b9665752-H--
Message: collections_remove_stale: Failed deleting collection (name "ip", key "45.72.81.9_ceaaa31b2410bc33155ef431d11705603ebb5f61"): Internal error (specific information not available)
Apache-Error: [file "apache2_util.c"] [line 271] [level 3] [client 172.58.197.123] ModSecurity: collections_remove_stale: Failed deleting collection (name "ip", key "45.72.81.9_ceaaa31b2410bc33155ef431d11705603ebb5f61"): Internal error (specific information not available)

I'm running
Ubuntu 20.04.4 LTS
Plesk Obsidian Version 18.0.42 Update #1, last updated on Mar 15, 2022 06:25 AM

using the plesk packaged versions of modsec, but the errors seem to be ones that were patched in 2018 at ModSecurity: collections_remove_stale: Failed deleting collection · Issue #576 · SpiderLabs/ModSecurity

# apt list --installed | egrep modsec
libapache2-modsecurity-plesk/focal,now 2.9.5-v.ubuntu.20.04+p18.0.42.0+t220117.1118 amd64 [installed]
plesk-modsecurity-configurator/now 18.0-v.ubuntu.20.04+p18.0.43.0+t220310.0712 all [installed,local]
plesk-modsecurity-crs/focal,now 1:3.3.2-v.ubuntu.20.04+p18.0.42.0+t220117.1118 amd64 [installed]

my /var/cache/modsecurity/www-data-ip.pag file had grown to 18GB.
This error would then make apache fail to recycle worker threads that would be stuck in the "L" (loggging) status in the apache status scorecard. Eventually all those would be exhausted and apache wouldn't accept new connections. Restarting apache would clear the connections and webpages would load again.



I see there is now a Modsec 3 for nginx available, but it doesn't yet support the Comodo ruleset. As we host mainly WordPress sites, the OWASP ruleset warning is quite offputting.

OWASP ModSecurity Core Rule Set is very restrictive and might block some functions (for example, file sharing, webmail) and some features of web applications (for example, WordPress plugins).

I may end up using OP's cron workaround for now to manually delete the cache at intervals to avoid availability issues when apache can't reuse workers stuck in logging status.

 

[Bitpalast]

I can confirm the issue described by @tmuka . We've seen the same last year. It got so bad that we had to switch to another ruleset. The problem is that once the page file is growing too large, it only takes seconds before apache gets stuck while driving the load up. It is a real blocker.

 

[websavers]

Weird, since the mod_security binary now installed from the Plesk repo is 2.9.5. Maybe the issue all along was twofold: the bug formerly identified a few years ago AND an actual issue with the way the comodo ruleset keys or accesses those entries.

Like Peter, we also switched rulesets a couple years ago and haven't had the issue since.

 

[tmuka]

Thanks for the responses. Are you using the OWASP or something else? We used to subscribe to the premium Atomic ruleset before the update to Ubuntu 20 when it wasn't supported in plesk.

 

[websavers]

Thanks for the responses. Are you using the OWASP or something else? We used to subscribe to the premium Atomic ruleset before the update to Ubuntu 20 when it wasn't supported in plesk.

We've been using Imunify360's ruleset, which is updated super-fast with protections for the latest web app vulnerabilities. Plus it's got a feature that disables rules for web apps that don't match what's installed on the domain to keep memory usage lightweight (runs nightly to determine which rules should apply for eachd omain), which is pretty handy! This issue doesn't occur with this ruleset.