serious CPU Load (kind of urgent trhead)

[MailParking]

since a couple of days, i've been monitoring a huge CPU Load wich was not normail before.

It has reached 150 couple of times, but usually rises up to 30 o 35.

I;ve figured out that if i stop apache or qmail, it goes down to 4, maybe 6 but i did not find an answer to this really.
Now, that is early here, servers seems to work fine, but i have even stop spamassassin, because it takes de load too high and other services start working slow. I am attaching results from top. Hope someone has something to share related to this.

here's some info:
top - 08:51:05 up 16:22, 1 user, load average: 1.65, 2.59, 3.22
Tasks: 138 total, 2 running, 133 sleeping, 0 stopped, 3 zombie
Cpu(s): 75.5% us, 24.2% sy, 0.0% ni, 0.0% id, 0.0% wa, 0.3% hi, 0.0% si
Mem: 1035300k total, 329068k used, 706232k free, 9304k buffers
Swap: 1044216k total, 289960k used, 754256k free, 39776k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
8083 root 25 0 19308 2112 2012 R 81.3 0.2 740:59.02 perl
27394 apache 15 0 74236 21m 6876 S 10.9 2.2 0:07.67 httpd
19995 popuser 18 0 5356 956 752 S 2.0 0.1 0:00.06 imapd
11425 named 15 0 39324 3324 1580 S 1.0 0.3 11:44.25 named
190 root 15 0 0 0 0 S 0.3 0.0 0:21.32 kjournald
2151 root 15 0 1632 556 488 S 0.3 0.1 0:16.36 syslogd
32221 mysql 15 0 132m 10m 3292 S 0.3 1.0 3:27.48 mysqld
19212 root 16 0 3876 992 740 R 0.3 0.1 0:00.64 top
1 root 16 0 2104 472 440 S 0.0 0.0 0:00.82 init
2 root 34 19 0 0 0 S 0.0 0.0 0:00.08 ksoftirqd/0
3 root 5 -10 0 0 0 S 0.0 0.0 0:00.04 events/0
4 root 5 -10 0 0 0 S 0.0 0.0 0:00.00 khelper
5 root 15 -10 0 0 0 S 0.0 0.0 0:00.00 kacpid
24 root 5 -10 0 0 0 S 0.0 0.0 0:00.00 kblockd/0
25 root 15 0 0 0 0 S 0.0 0.0 0:00.00 khubd
34 root 15 0 0 0 0 S 0.0 0.0 0:00.00 pdflush
35 root 15 0 0 0 0 S 0.0 0.0 0:02.53 pdflush
37 root 7 -10 0 0 0 S 0.0 0.0 0:00.00 aio/0
36 root 16 0 0 0 0 S 0.0 0.0 1:13.88 kswapd0
111 root 25 0 0 0 0 S 0.0 0.0 0:00.00 kseriod
181 root 6 -10 0 0 0 S 0.0 0.0 0:00.00 ata/0
183 root 22 0 0 0 0 S 0.0 0.0 0:00.00 scsi_eh_0
184 root 23 0 0 0 0 S 0.0 0.0 0:00.00 scsi_eh_1
1046 root 6 -10 1804 348 344 S 0.0 0.0 0:00.01 udevd
1440 root 6 -10 0 0 0 S 0.0 0.0 0:00.00 kauditd
1473 root 6 -10 0 0 0 S 0.0 0.0 0:00.00 kmirrord
1474 root 7 -10 0 0 0 S 0.0 0.0 0:00.00 kmir_mon
1498 root 23 0 0 0 0 S 0.0 0.0 0:00.00 kjournald
2155 root 16 0 2868 384 380 S 0.0 0.0 0:00.00 klogd
2182 rpc 15 0 3280 416 412 S 0.0 0.0 0:00.00 portmap
2201 rpcuser 18 0 2616 560 556 S 0.0 0.1 0:00.00 rpc.statd
2227 root 16 0 5824 760 728 S 0.0 0.1 0:00.02 rpc.idmapd
2298 root 17 0 3312 556 432 S 0.0 0.1 0:00.01 smartd
2307 root 21 0 3112 416 412 S 0.0 0.0 0:00.00 acpid
2316 root 16 0 8988 1172 1096 S 0.0 0.1 0:00.11 cupsd
2396 root 16 0 4904 1108 1012 S 0.0 0.1 0:00.77 sshd
2687 root 16 0 4824 772 724 S 0.0 0.1 0:00.25 couriertcpd
2700 root 15 0 5208 728 724 S 0.0 0.1 0:00.00 couriertcpd
2702 root 15 0 4624 512 508 S 0.0 0.0 0:00.00 courierlogger
2710 root 15 0 5000 776 724 S 0.0 0.1 0:02.77 couriertcpd
2712 root 15 0 3976 560 508 S 0.0 0.1 0:02.62 courierlogger
2721 root 25 0 3720 720 716 S 0.0 0.1 0:00.00 couriertcpd
2723 root 25 0 5176 328 324 S 0.0 0.0 0:00.00 courierlogger
2733 root 19 0 2944 416 412 S 0.0 0.0 0:00.00 gpm
3873 root 16 0 49040 1108 1048 S 0.0 0.1 0:00.13 httpsd
3903 psaadm 16 0 56220 14m 10m S 0.0 1.5 0:08.16 httpsd
3904 psaadm 16 0 67872 11m 8012 S 0.0 1.1 0:13.06 httpsd
3905 psaadm 16 0 57836 6508 4712 S 0.0 0.6 0:05.74 httpsd
3906 psaadm 16 0 55600 10m 7008 S 0.0 1.0 0:05.12 httpsd
3907 psaadm 15 0 56604 6688 4768 S 0.0 0.6 0:06.73 httpsd
4005 psaadm 15 0 54520 6032 4492 S 0.0 0.6 0:03.13 httpsd
4007 psaadm 16 0 59652 14m 9.8m S 0.0 1.4 0:11.88 httpsd
4008 psaadm 16 0 68468 10m 6944 S 0.0 1.0 0:06.97 httpsd
4332 psaadm 16 0 59144 17m 11m S 0.0 1.7 0:06.91 httpsd
5844 htt 18 0 2572 256 252 S 0.0 0.0 0:00.00 htt

 

[MailParking]

More information

I've stoped:
Dr.Web
SpamAssassin
PostgreSQL
Tomcat
And left running:
Apache
QMail
Courier-IMAP
BIND

Load seems to be stable (Load avg: between 2.x and 3.x) but sometimes goes to +15 and i can control it stoping qmail or apache until it goes down and i restart them and seems stable again.

Heres the top info when going up.
Only think i can see is Perl using a los of CPU%

top - 15:01:34 up 22:33, 1 user, load average: 23.81, 14.59, 8.01
Tasks: 162 total, 2 running, 152 sleeping, 0 stopped, 8 zombie
Cpu(s): 74.1% us, 25.2% sy, 0.0% ni, 0.0% id, 0.0% wa, 0.7% hi, 0.0% si
Mem: 1035300k total, 294008k used, 741292k free, 1732k buffers
Swap: 1044216k total, 163024k used, 881192k free, 28196k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
8083 root 25 0 19308 1852 1752 R 90.2 0.2 1053:04 perl
2547 root 17 0 16128 5108 2956 D 1.3 0.5 0:00.08 php
11425 named 16 0 39324 2872 1504 S 1.0 0.3 14:38.07 named
1444 apache 17 0 64888 22m 6632 D 1.0 2.2 0:00.33 httpd
30415 apache 15 0 69836 26m 7688 S 0.7 2.6 0:30.27 httpd
16243 root 16 0 3944 924 676 R 0.7 0.1 0:13.71 top
2546 barloven 17 0 7564 5984 1896 D 0.7 0.6 0:00.41 GET
2635 popuser 17 0 6456 932 748 D 0.7 0.1 0:00.02 imapd
36 root 15 0 0 0 0 S 0.3 0.0 1:49.99 kswapd0
32221 mysql 15 0 139m 9352 3148 S 0.3 0.9 5:44.12 mysqld
1026 root 15 0 17216 2868 1788 S 0.3 0.3 0:00.63 wdcollect
31514 apache 15 0 69908 26m 7496 S 0.3 2.6 0:24.30 httpd
2773 root 16 0 5384 1236 872 D 0.3 0.1 0:00.01 authpsa
1 root 15 0 2104 452 420 S 0.0 0.0 0:00.87 init
2 root 34 19 0 0 0 S 0.0 0.0 0:00.08 ksoftirqd/0
3 root 5 -10 0 0 0 S 0.0 0.0 0:00.12 events/0
4 root 5 -10 0 0 0 S 0.0 0.0 0:00.00 khelper
5 root 15 -10 0 0 0 S 0.0 0.0 0:00.00 kacpid
24 root 5 -10 0 0 0 S 0.0 0.0 0:00.00 kblockd/0
25 root 15 0 0 0 0 S 0.0 0.0 0:00.00 khubd
34 root 15 0 0 0 0 S 0.0 0.0 0:00.00 pdflush
35 root 15 0 0 0 0 S 0.0 0.0 0:03.66 pdflush
37 root 7 -10 0 0 0 S 0.0 0.0 0:00.00 aio/0
111 root 25 0 0 0 0 S 0.0 0.0 0:00.00 kseriod
181 root 6 -10 0 0 0 S 0.0 0.0 0:00.00 ata/0
183 root 22 0 0 0 0 S 0.0 0.0 0:00.00 scsi_eh_0
184 root 23 0 0 0 0 S 0.0 0.0 0:00.00 scsi_eh_1
190 root 15 0 0 0 0 D 0.0 0.0 0:30.04 kjournald
1046 root 6 -10 1804 324 324 S 0.0 0.0 0:00.01 udevd
1440 root 6 -10 0 0 0 S 0.0 0.0 0:00.00 kauditd
1473 root 6 -10 0 0 0 S 0.0 0.0 0:00.00 kmirrord
1474 root 7 -10 0 0 0 S 0.0 0.0 0:00.00 kmir_mon
1498 root 23 0 0 0 0 S 0.0 0.0 0:00.00 kjournald
2151 root 15 0 1632 536 488 D 0.0 0.1 0:24.71 syslogd
2155 root 16 0 2868 380 380 S 0.0 0.0 0:00.00 klogd
2182 rpc 15 0 3280 412 412 S 0.0 0.0 0:00.00 portmap
2201 rpcuser 18 0 2616 556 556 S 0.0 0.1 0:00.00 rpc.statd
2227 root 16 0 5824 756 728 S 0.0 0.1 0:00.02 rpc.idmapd
2298 root 17 0 3312 592 440 S 0.0 0.1 0:00.02 smartd
2307 root 21 0 3112 408 408 S 0.0 0.0 0:00.00 acpid
2316 root 15 0 8988 1168 1096 S 0.0 0.1 0:00.18 cupsd
2396 root 16 0 4904 996 996 S 0.0 0.1 0:00.97 sshd
2687 root 16 0 4824 768 724 S 0.0 0.1 0:00.80 couriertcpd
2691 root 15 0 3848 564 508 S 0.0 0.1 0:00.91 courierlogger
2700 root 15 0 5208 736 724 S 0.0 0.1 0:00.00 couriertcpd
2702 root 16 0 4624 544 508 S 0.0 0.1 0:00.00 courierlogger
2710 root 16 0 5000 776 724 S 0.0 0.1 0:06.23 couriertcpd
2712 root 16 0 3976 560 508 S 0.0 0.1 0:06.03 courierlogger
2721 root 15 0 3720 728 724 S 0.0 0.1 0:00.00 couriertcpd
2723 root 16 0 5308 540 508 S 0.0 0.1 0:00.00 courierlogger
2733 root 19 0 2944 412 412 S 0.0 0.0 0:00.00 gpm
5844 htt 18 0 2572 252 252 S 0.0 0.0 0:00.00 htt
5845 htt 17 0 8920 1032 1032 S 0.0 0.1 0:00.00 htt_server
5886 canna 16 0 19684 500 476 S 0.0 0.0 0:00.66 cannaserver
6173 root 15 0 4632 696 624 S 0.0 0.1 0:00.45 crond

 

[jens.jensen]

try doing a "ps -aux" to find out where that offending perl script is being called from

 

[MailParking]

Thank you for your reply.

I was able to finish the Perl process and no problems there BUT i realized that if i restart spamassassin, CPU load increases to +25! So i ahve controlled performance stopping spamassassin, but this is not a very full solution.
Any idea?

 

[Magumo]

Hi Julian,

I also have similiar high CPU due to spamassassin. The moment I disable it the load stabilizes to around 2 and upon enabling it it shoots to above 10.

My main problem now is running without it is resulting lots of spam for my users. I would like to be able to run it but efficiently without causing slowness all the users.