Server hangs, Plesk reports extremely high CPU usage

[totalidea]

We are observing something that we were not able to fix yet on our Centos 6.3 server with Plesk 11 (latest patch -11.0.9 #13) installed: Out of a sudden our server hangs. After reboot, Plesk reports an extremely high CPU usage (beyond million %). Sometimes this happens 2-3 times a day, sometimes it works fine for 24 hours, sometimes for a week before it happens again. No regularity in this: it happens at different times. When connected to the server via serial console, top does not report a high CPU usage, neither before the crash, nor after rebooting.

We have checked all possible log files, but there's nothing suspicious, and nothing we can image that could somehow give us a hint. At least we have not found something...

I have read http://forum.parallels.com/showthread.php?t=260744 but nothing of the included tips helps. I have also boogled up and down, but nothing.

Our provider ran a memory and a stress test: no result. rkhunter: nothing.

Finally I thought that someone here could give me a hint what the cause may be. Thanks very much in advance.

Steve

 

[abdi]

What service does plesk report to use that much CPU? Also, may I suggest you install cpulimit it will help you mitigate the server hangs.

Installation instructions are:

To limit the CPU usage of httpd and php, we do the following:

wget http://downloads.sourceforge.net/cpulimit/cpulimit-1.1.tar.gz
tar xvfz cpulimit-1.1.tar.gz
cd cpulimit-1.1/
make

We create a file called cpu and the contents below:

vim cpu

add the contents below for the service you want to limit

./cpulimit -P /usr/bin/php -l 20
./cpulimit -e httpd -l 20

Then leave it running by...

sh cpu &

 

[totalidea]

Thanks for the reply and the suggestions. I'll try that asap.

Plesk reports Apache with the extreme CPU usage. Additionally, the network card shows an extreme bandwidth, as well. I forgot to mention that before.

 

[abdi]

If its apache then most likely you are under a DOS attack during that time. Implement CPULIMIT instructions above and limit apache such that in cases as those, instead of it bringing down your server, only it will be killed!

 

[totalidea]

OK, I have it up and running, but when executing

sh cpu &

it returns

[1] 2223
[XXXXX@XXXXX]# Warning: no target process found. Waiting for it...

Does that mean that it cannot find the httpd process?

 

[abdi]

That is what it means, please check the name of your apache process on your server. Is it httpd or apache2

 

[totalidea]

OK, running. However, once I terminate the terminal: what then?

 

[abdi]

if you started it with sh cpu & then it will continue running!

 

[totalidea]

Thanks very much! Is there any way to test the whole thing?

 

[totalidea]

And another question:

httpd only appears in

top

when a http request is actually coming in. Can that be the reason why cpuusage reports that it cannot find a target process and that it waits for it? If so, does cpuusage sit there once started and wait for httpd to appear, then limit it in case it requests more than 20%?

I really appreciate your help.

 

[abdi]

Yes you can test by causing a lot of traffic to your website / server and see how it reacts to it. You can run the following:

/usr/bin/ab -c 50 -n 10000 http://www.yourdomain.com/

usr/bin/ab --help for further help!

 

[totalidea]

Brilliant. Test succeeded. 100.000 requests - nothing beyond 20%.

We thought that it could be a DOS attack, but could not imagine. Thanks so much! You made my day!

 

[abdi]

Cool, and your most welcome!

 

[GouravJ]

I am also getting same problem with our linux server but in our case it shows that mysql is consuming the resource, can u please help.