100% CPU usage by httpd process.

[RomanF]

Hello.
I have strange problem with Apache and Plesk 8.6. When somebody check mailbox using webmail (based on psa-horde), httpd daemon "freeze" and then start using 100% CPU. Any "freeze" httpd process has an unclosed connection to imap port (It can be shown by lsof command.). Can somebody explain why httpd don't close it after user logout? Also, I check hardware, include several RAM tests, and looks, like hardware is ok.. Maybe I forgot to check something else? Thanks in advance, Roman.

OS (Fedora8):
Linux hosting-node-0 2.6.26.8-57.fc8 #1 SMP Thu Dec 18 18:59:49 EST 2008 x86_64 x86_64 x86_64 GNU/Linux
Installed packages:
httpd-2.2.9-1.fc8
php-imap-5.2.6-2.fc8
psa-courier-imap-add-8.6.0-fc8.build86080930.03
psa-horde-3.1.7-fc8.build86080722.00
Selinux was disabled at boot.

Here is verbose output:

[u1@hosting-node-0 u1]$ top -d1 | head -n 10
top - 07:25:34 up 23:39, 1 user, load average: 4.28, 3.68, 3.70
Tasks: 190 total, 4 running, 186 sleeping, 0 stopped, 0 zombie
Cpu(s): 61.0%us, 1.5%sy, 0.6%ni, 32.8%id, 3.9%wa, 0.0%hi, 0.1%si, 0.0%st
Mem: 4062748k total, 3251332k used, 811416k free, 271976k buffers
Swap: 3911788k total, 6196k used, 3905592k free, 1879112k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1104 apache 20 0 427m 42m 6388 R 100 1.1 1097:31 httpd
22940 apache 20 0 423m 43m 5408 R 100 1.1 774:31 httpd
9856 apache 20 0 425m 39m 5896 R 100 1.0 1370:10 httpd

[u1@hosting-node-0 base]$sudo lsof -p 1104
... Skipped ...
httpd 1104 apache 196uW REG 8,3 63655 3811934 /var/lib/php/session/sess_094u0rt3a6imbdmil2f8khkjh6 (deleted)
httpd 1104 apache 197u unix 0xffff8100726e6c00 562283 socket
httpd 1104 apache 198u IPv4 562285 TCP hosting-node-0.mydomain.org:42407->hosting-node-0.mydomain.org:imap (CLOSE_WAIT)

 

[IgorG]

It is difficult to say what may be wrong there with provided information... Not sure that it is imap related issue, seems something wrong with apache or maybe some wrong custom script... I can suggest you try to strace apache processes. Most probably that it will help to find a reason of this problem.

 

[ishipenkov]

I can suggest you try to strace apache processes. Most probably that it will help to find a reason of this problem.

We try to strace apache processes and it shows nothing with freezed process. Totally nothing - there are no messages at all.

PS: I'm a colleague of a RomanF

 

[RomanF]

Apache strace.

I run strace on normal Apache process and handle a moment when Apache "freeze". After this moment strace show nothing. Strace output is in included file. I can't find any abnormal things in strace output. Any suggestions?

 

[Hultenius]

I just had the same problem. Two of four processor cores where blocked by two httpd processes. I was able to trace both processes to vhost "horde.webmail" thanks to the apache "server-status" page. This usefull page also gave me the IP of the user which made it easy to lookup the httpd requests in /var/log/httpd/access_log.

Unfortunately, there was nothing strange there (well, nothing obvious). The user where running Horde over https with web browser Firefox 3.5.6.

System:
CentOS 5.4 (64-bit)
Plesk 9.2.3
Apache 2.2.3
PHP 5.2.11
Atomic Secured Linux

# lsof -p 32716
...
httpd 32716 root 388u REG 8,1 0 79855678 /tmp/ZCUD1jfKHI (deleted)
httpd 32716 root 389u 0000 0,7 0 91 unknown inode type
httpd 32716 root 390u IPv4 12621738 TCP {SERVER_HOSTNAME}:https->{USER_HOSTNAME}:lv-ffx (CLOSE_WAIT)
httpd 32716 root 391u REG 8,1 79192 232196439 /var/lib/php/session/sess_edtbl82m3ckjh954leilgb2016 (deleted)

 

[IgorG]

I think that it is complex problem and deep investigation directly on server is required. Not sure that such problems can be solved as forum's discussion. Therefore I can suggest you contact Support Team regarding this problem. It will be investigated and fixed much more effectively.

 

[TomWork]

Hey guys,

We are encountering exactly the same issue : empty strace, httpd hogging the CPU. So what was the solution ? I have this feeling it is linked to php-imap because we have this problem with our webmail as well (roundcube).

Note: That's funny, we have the same problem a year later, to the day. And sorry to dig up this thread but I couldn't find anything else.

virtuozzo-release-4.0.0-520

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
14537 apache 25 0 155m 14m 9984 R 32.3 0.1 48:26.95 httpd
15179 apache 25 0 157m 19m 11m R 31.6 0.1 62:38.29 httpd
15185 apache 25 0 154m 14m 9.9m R 31.6 0.1 61:54.24 httpd