1. Please take a little time for this simple survey! Thank you for participating!
    Dismiss Notice
  2. Dear Pleskians, please read this carefully! New attachments and other rules Thank you!
    Dismiss Notice
  3. Dear Pleskians, I really hope that you will share your opinion in this Special topic for chatter about Plesk in the Clouds. Thank you!
    Dismiss Notice

Issue Mod_Security IP persistent storage massive

Discussion in 'Plesk Onyx for Linux' started by websavers, Jun 23, 2017.

  1. websavers

    websavers Basic Pleskian

    16
     
    Joined:
    Jul 15, 2013
    Messages:
    90
    Likes Received:
    14
    This issue affects CentOS6 running mod_security-2.9.0-centos6.16102616.x86_64
    From repo : PLESK_17_0_17-extras

    OS ‪CentOS 6.9 (Final)‬
    Product Plesk Onyx
    Version 17.0.17 Update #28, last updated on June 23, 2017 04:09 AM

    The detected symptoms of the issue:
    - High CPU usage on servers with lots of traffic or at least lots of domains. It's particularly noticeable as IO load if you're watching iotop or htop.
    - The file /var/cache/modsecurity/ip.pag grows to be over 1GB in size

    Like any apache related log file, this means heavy IO load while regularly reading and appending to this database file.

    This appears to be a known (and solved) issue as described in the ModSec GitHub repo here. At the bottom of that GitHub issue is a marked solution as found here.

    I'm hoping you folks might be so kind as to apply the patch/solution to the build of ModSec in the Plesk 17 extras repo to resolve this issue. Note that I think 2.9.1 doesn't even have this patch applied yet, and so it would need to be manually applied to your SRPMs in the repo.

    At the moment our only usable workaround to keep the performance of our CentOS6 boxes in check is to zero out the database file every day, which isn't great for the effectiveness of Mod_Security when it's IP persistence is important for IP banning!

    Thanks in advance for any help that can be provided on this issue.
     
    Last edited: Jun 24, 2017
  2. surfgatinho

    surfgatinho New Pleskian

    1
    70%
    Joined:
    Apr 6, 2017
    Messages:
    5
    Likes Received:
    0
    Location:
    UK
    Where you getting errors along the lines of "Message: collections_remove_stale: Failed deleting collection" with this issue?
    And did you manage to fix it?

    Also, which ruleset were you running?
     
  3. websavers

    websavers Basic Pleskian

    16
     
    Joined:
    Jul 15, 2013
    Messages:
    90
    Likes Received:
    14
    I'm honestly not sure if there were such errors; did you find those in the audit log or in the main apache error log?

    We're running the Comodo ruleset.

    I wouldn't say we fixed it, but we worked around it by having a cron job run daily like this: cat /dev/null > /var/cache/modsecurity/ip.dir && cat /dev/null > /var/cache/modsecurity/ip.pag. Clearly it's not optimal for security, but neither is the complete and utter destruction of performance on all of our busy servers.
     
    Alban Staehli likes this.
  4. surfgatinho

    surfgatinho New Pleskian

    1
    70%
    Joined:
    Apr 6, 2017
    Messages:
    5
    Likes Received:
    0
    Location:
    UK
    I found the errors in both the audit log and the apache logs.

    This only started happening to me when I changed over to the Comodo ruleset. When I use the OSWAP ruleset it bans practically everything but the ip.pag file doesn't grow.

    Makes me wonder if the issue is with the Comodo ruleset. I might try OSWAP again but introduce it one rule at a time...
     
  5. websavers

    websavers Basic Pleskian

    16
     
    Joined:
    Jul 15, 2013
    Messages:
    90
    Likes Received:
    14
    According to the bug report I linked to on GitHub, the fix for this issue comes out of the persistent storage collection name not being passed properly through to the storage DB. In other words, my bet is on OWASP not naming their collection and thusly using default_ as the collection prefix (which works fine) vs. Comodo naming their collection (perhaps something like comodo_) which doesn't work because of that bug.

    Technically Comodo would be doing it according to spec (the right way) but that bug causes the problem. If my guess is correct, then it's still up to Plesk to include that patch in their build of mod_security. Once fixed, then Comodo's namespace would no longer suffer from the bug and therefore removal of records from persistent storage would proceed as it's expected to.
     
    pleskpanel likes this.
  6. Alban Staehli

    Alban Staehli Basic Pleskian

    23
    40%
    Joined:
    Mar 8, 2010
    Messages:
    96
    Likes Received:
    0
    Same issue here with Centos 7.4 - after a couple of weeks, /var/cache/modsecurity/ip.pag size is 12G !!! Clearly an issue when using Comodo rules.
    thx for the daily script to clear the /var/cache/modsecurity/ip.pag file.
     
  7. UFHH01

    UFHH01 Plesk addicted!

    44
    64%
    Joined:
    Jun 11, 2013
    Messages:
    6,762
    Likes Received:
    1,709
    Location:
    Hamburg / Germany
  8. Alban Staehli

    Alban Staehli Basic Pleskian

    23
    40%
    Joined:
    Mar 8, 2010
    Messages:
    96
    Likes Received:
    0
    Hi @UFHH01 ,
    Thx for your feedback - I've even commented the post in regards to Centos 7.4 last Monday as I did a successful upgrade.
    But, please, note that the size of /var/cache/modsecurity/ip.pag was already huge prior to last Monday - I checked a backup. Therefore, it's not a bug related to Centos 7.4, but as mentioned by @websavers , the issue seems to be more related to the use of Comodo rules and ModSec under Plesk.
    Cheers
     
  9. UFHH01

    UFHH01 Plesk addicted!

    44
    64%
    Joined:
    Jun 11, 2013
    Messages:
    6,762
    Likes Received:
    1,709
    Location:
    Hamburg / Germany
    Hi Alban Staehli,

    To report a ( possible ) bug at Plesk, its components or extensions, you should consider to open a new bug - report at:

     
  10. Alban Staehli

    Alban Staehli Basic Pleskian

    23
    40%
    Joined:
    Mar 8, 2010
    Messages:
    96
    Likes Received:
    0
Loading...