apache runnning out of memory and swap space

[KudosDesigns]

Hi,

We have had major issues with memory and linux running out of swap space. This has caused our servers to completely hang and not process requests.

We are running 150 domains on a P4 3.0Ghz machine with 4Gb RAM and 2Gb SWAP. We process around 10K messages per day with qmail and run around 75 domains with PHP scripted content managed websites.

The machine seems to be using swap space as httpd runs out of memory. We have stabilised the situation by implementing two things to httpd.conf (Apache httpd-2.0.54-10.4 on FC4):

MaxRequestsPerChild 200
RLimitMEM 100663296 1610612736

This should mean Apache processes are re-spawed after 200 requests (rather than 4000) and memory to Apache is top limited to 1.5Gb and each process can only consume 96Mb (we are running 16 httpd processes usually).

Although this has stopped memory running away with itself, we are very concerned that Apache isn't managing this itself. I have heard that PHP 5 shouldn't really be ever run on Apache 2 - is this the reason? If you restart apache gracefully, memory use goes down to dramatically - it is as if Apache isn't giving up its memory properly.

Does anyone know how to control memory on Apache without manually setting it like above? Should we upgrade to FC5 and run Apache 2.22 - would this help? Is it something to do with MPM and PHP - how do i make it safe as the PHP website recommends?

Or maybe we just need more RAM - but we did have 2Gb in the machine until this morning, upgraded it to 4Gb and it still fell over under the same load.

Thanks

Nick

To add to this, I have found out that we are indeed running prefork.c in our Apache version:

apachectl -l
core.c
prefork.c
http_core.c
mod_so.c

and here is our configuration parameters:

<IfModule prefork.c>
StartServers 8
MinSpareServers 5
MaxSpareServers 20
ServerLimit 256
MaxClients 256
MaxRequestsPerChild 200
</IfModule>

Maybe we should low MaxClients???? Help!

 

[atomicturtle]

are you using mod_security, or mod_rewrite rules at all? Either one can cause this kind of memory leak to happen.

Whoever told you that php5 had issues with apache 2 must have been reading cpanel press releases.

 

[KudosDesigns]

The PHP5 and Apache thing does seem to be a red herring after reading further.

However, we do extensively use mod_security with rules from gotroot.com and mod_rewrite is used extensively on all our content managed websites.

We have it under control now with the "fixes" in the apache configuration ... but i would be happier if i could get it working with the default configuration.

mod_security is excellent and has stopped security breaches on our site no end.

We use:

mod_security-1.9.4-1.fc4

Rules are from gotroot.com (slightly tweaked for our setup) and are updated daily.

Any ideas how to get to the bottom of the memory problem?

 

[atomicturtle]

Is this using the blacklists as well? I'd be interested seeing the exact rulesets you're using.

What you've done so far is a good place to start on a long term fix. I'd always suspected that the problem was mod_security wasnt cleaning up after itself. Forcing it to behave through an apache directive might not be the perfect fix, but its certainly getting the module to behave a little better.

We definitely dont have to do anything like this on rh9, although one thing to point out is that the default MaxRequsts on that is 1000. That almost made me think that was the sole fix, if it wasnt for the fact that RHEL3/CentOS3 use that as well.

 

[KudosDesigns]

Yes we are using the blacklist.conf and blacklist2.conf.

We are using the rules rules as exactly from gotroot.com including the files that you download as part of the 1.9.4 rule set.

The only bits we have altered are the exclude rules as and when we come across rules that are false positives.

We don't include:

recons.conf and badips.conf

We update the rules daily using the script on gotroot.com by Ole Martin Eide. This seems to work fine.

Is it the blacklist.conf file that causes it to break? Should i just remove that?

Presumably though, removing that would cause it not to blacklist known spammer sites which is a useful feature.

Annoyingly i have tried to re-create the situation using ab (Apache Bench) on an identical server with the same rule sets and my patch up configuration Apache memory fix not present. Even when i up the load massively, i can't get the box to fall over and start swapping ... I guess I would have to simulate the exact same traffic from the logs as caused the box to fail rather than just the repeatedly asking for content at the same URL.

Would mod_security 2.0 help? Does anyone know the exact rule set that is causing this?

 

[KudosDesigns]

I can pass you on our mod_security rule set with which we are getting memory leaks. I would prefer not to post them to the reply on the forum as i'd prefer not to blurt out our security config to everyone in the world - send me a PM with your mail address and i'll send them through. We would obviously be more than happy to share a solution though! - it would make me happy for one !

If you want me to help debugging the memory leak so other people don't have the same issue, i would be more than happy to help - just let me know if I can be of further assistance.

 

[atomicturtle]

We discontinued badips.conf anyway, so thats a good thing that you're not using it. And yes, the blacklists are what use to cause the memory leak, just because those are the largest rulesets. Mike and I were just discussing what you've run across, vs. some observations in the apache rpms used after RH9.

As you've noticed, one of the problems is that this is so damn hard to reproduce.

 

[KudosDesigns]

I am going to disable blacklist.conf then from my mod_security setup and see what happens.

I will report back if it all works - or falls in a massive heap! ...

For information, a memory leak of sorts occurred at around 16:00 today - i checked the mod security audit logs and one site was blocked at 15:45 and on at 16:15 - it wasn't as if there was a high amount of users being rejected ... so I am still non the wiser as to what actual combination of activities is causing it (so i could attempt to replicate it).

 

[KudosDesigns]

The memory leak was definitely casued by mod_security and the blacklist.conf file. It has been runnning for a week and a half now with no memory problems whatsoever, and not a hint of swap file usage.

For reference, my setup is:
Plesk 8.0.1
Fedora Core 4
Single Processor (3Ghz)
Apache 2.0.54
2Gb RAM
mod_security 1.9.4 (Fedora Core extras release)
Ruleset from gotroot.com

This reliably does not work with blacklist.conf enabled. If you disable it, the memory leaks stop. You can help the leaks by altering specific Apache config settings relating to memory, although it doesn't go away 100% (it's just more controlled - but under high load you would most likely experience swap file action).

I imagine that atmoicrocketturtle or someone from gotroot.com will let us all know when mod_security or Apache or the ruleset is fixed so that we can use blacklists again ... until then they are disabled for me.

 

[atomicturtle]

We've got an alternate implementation for the blacklists now by moving them off into an RBL/SURBL. The intent of that rule set was to block comment spam, and in the research I was doing with this, mike and I came up with a much more scalable design. We want to keep the rules associated with security loaded in memory, and the ones associated with spam in some place less time-sensitive. Ergo, your users arent going to complain if a POST in a forum takes an extra second or two, vs front loading all that into consuming a crazy amount of memory.

 

[KudosDesigns]

Does that mean that RBL and mod security 2.1 should work ok without memory leaks?

 

[atomicturtle]

Yeah, it moves the rules off of the server and into a network based distribution model. The down side is that it effects performance, so you have to do it in the right place so it doesn't kill the box.