CPU Load

[kram@]

Hello All,

For the past couple of days i have been seeing a 99% CPU rate. The only info i have is that is is a perl process.

Anybody have a clue on how to find out what it is???

PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND
15689 apache 25 0 3404 3300 1436 R 21.9 0.1 20:52 0 perl
20357 apache 25 0 3076 3072 1112 R 20.7 0.1 26:06 0 perl
21243 apache 20 0 3068 3068 1104 R 20.1 0.1 24:15 0 perl
21574 apache 25 0 3076 2852 1112 R 19.9 0.1 25:33 0 perl

 

[wagnerch]

You probably have a vulnerable web application, and now spammers have injected applications that are probably firing off spam, or running some sort of service.

Check your qmail message stats:

/var/qmail/bin/qmail-qstat


Then read the thread below for some tips:

http://forum.swsoft.com/showthread.php?s=&threadid=30978&highlight=readlink

 

[jplotzke]

I've had the exact same problem -- 99% CPU by perl scripts. It turned out to be hackers sending spam through my box.

I'd follow the link that wagnerch suggested and follow that. In the mean time, you can kill those perl processes, but they'll probably start up again soon if they're being triggered. In my case, all the attack scripts were in /tmp and /var/tmp. You may want to check those folders for anything suspicious.

~Jeff

 

[kram@]

Hello,

I ran the /var/qmail/bin/qmail-qstat
Nothing out of the norm here, only 90 msg in the que.

Folllowed the other links and did:

ps -fuapache

readlink /proc/22695/exe
/usr/bin/perl

This is the same for all PID??

UID PID PPID C STIME TTY TIME CMD
apache 22695 1 0 Jul16 ? 00:02:30 httpssql
apache 22751 1 0 Jul16 ? 00:03:16 httpssql
apache 22898 1 0 Jul16 ? 00:02:13 httpssql
apache 22930 1 0 Jul16 ? 00:03:01 httpssql
apache 22963 1 0 Jul16 ? 00:03:05 httpssql
apache 23026 1 0 Jul16 ? 00:02:36 httpssql
apache 23039 1 0 Jul16 ? 00:03:59 httpssql
apache 23053 1 0 Jul16 ? 00:02:36 httpssql
apache 23724 1 0 Jul16 ? 00:02:01 httpssql
apache 23734 1 0 Jul16 ? 00:02:38 httpssql
apache 23773 1 0 Jul16 ? 00:03:17 httpssql
apache 23842 1 0 Jul16 ? 00:03:59 httpssql
apache 23859 1 0 Jul16 ? 00:03:18 httpssql
apache 23870 1 0 Jul16 ? 00:03:19 httpssql
apache 23885 1 0 Jul16 ? 00:03:11 httpssql
apache 23933 1 0 Jul16 ? 00:02:42 httpssql
apache 23975 1 0 Jul16 ? 00:02:40 httpssql
apache 24054 1 0 Jul16 ? 00:02:47 httpssql
apache 24078 1 0 Jul16 ? 00:03:03 httpssql
apache 24104 1 0 Jul16 ? 00:04:48 httpssql
apache 24191 1 0 Jul16 ? 00:02:31 httpssql
apache 24806 1 0 Jul16 ? 00:03:23 httpssql
apache 24829 1 0 Jul16 ? 00:02:10 httpssql
apache 24853 1 0 Jul16 ? 00:02:15 httpssql
apache 24869 1 0 Jul16 ? 00:02:31 httpssql
apache 24895 1 0 Jul16 ? 00:02:38 httpssql
apache 24937 1 0 Jul16 ? 00:03:10 httpssql
apache 24969 1 0 Jul16 ? 00:03:00 httpssql
apache 24987 1 0 Jul16 ? 00:02:07 httpssql
apache 25087 1 0 Jul16 ? 00:03:12 httpssql
apache 25185 1 0 Jul16 ? 00:04:10 httpssql
apache 25193 1 0 Jul16 ? 00:03:15 httpssql
apache 25237 1 0 Jul16 ? 00:02:56 httpssql
apache 25255 1 0 Jul16 ? 00:02:53 httpssql
apache 25353 1 0 Jul16 ? 00:02:48 httpssql
apache 25368 1 0 Jul16 ? 00:01:59 httpssql
apache 25793 1 0 Jul16 ? 00:02:23 httpssql
apache 25838 1 0 Jul16 ? 00:02:08 httpssql
apache 25858 1 0 Jul16 ? 00:02:38 httpssql
apache 25867 1 0 Jul16 ? 00:02:23 httpssql
apache 25975 1 0 Jul16 ? 00:03:21 httpssql
apache 25989 1 0 Jul16 ? 00:02:52 httpssql
apache 26028 1 0 Jul16 ? 00:02:53 httpssql
apache 26076 1 0 Jul16 ? 00:02:15 httpssql
apache 26115 1 0 Jul16 ? 00:02:20 httpssql
apache 26226 1 0 Jul16 ? 00:02:27 httpssql
apache 26250 1 0 Jul16 ? 00:02:31 httpssql
apache 26266 1 0 Jul16 ? 00:02:24 httpssql
apache 26293 1 0 Jul16 ? 00:03:27 httpssql
apache 26320 1 0 Jul16 ? 00:02:25 httpssql
apache 26324 1 0 Jul16 ? 00:02:23 httpssql
apache 26385 1 0 Jul16 ? 00:03:24 httpssql
apache 26888 1 0 Jul16 ? 00:03:09 httpssql
apache 26912 1 0 Jul16 ? 00:02:07 httpssql
apache 26929 1 0 Jul16 ? 00:02:31 httpssql
apache 26968 1 0 Jul16 ? 00:02:28 httpssql
apache 26980 1 0 Jul16 ? 00:02:58 httpssql
apache 26989 1 0 Jul16 ? 00:02:15 httpssql
apache 27262 1 0 Jul16 ? 00:02:29 httpssql
apache 27314 1 0 Jul16 ? 00:02:02 httpssql
apache 27938 1 0 Jul16 ? 00:05:40 httpssql
apache 27940 1 0 Jul16 ? 00:03:44 httpssql
apache 27983 1 0 Jul16 ? 00:02:52 httpssql
apache 28065 1 0 Jul16 ? 00:02:10 httpssql
apache 28232 1 0 Jul16 ? 00:02:28 httpssql
apache 28255 1 0 Jul16 ? 00:02:33 httpssql
apache 28265 1 0 Jul16 ? 00:02:32 httpssql
apache 28315 1 0 Jul16 ? 00:02:46 httpssql
apache 28379 1 0 Jul16 ? 00:04:20 httpssql
apache 28400 1 0 Jul16 ? 00:02:38 httpssql
apache 28465 1 0 Jul16 ? 00:02:25 httpssql
apache 28469 1 0 Jul16 ? 00:02:24 httpssql
apache 28540 1 0 Jul16 ? 00:02:31 httpssql
apache 28570 1 0 Jul16 ? 00:03:17 httpssql
apache 29010 1 0 Jul16 ? 00:02:10 httpssql
apache 29090 1 0 Jul16 ? 00:02:10 httpssql
apache 29120 1 0 Jul16 ? 00:01:59 httpssql
apache 29173 1 0 Jul16 ? 00:02:11 httpssql
apache 29202 1 0 Jul16 ? 00:03:02 httpssql
apache 29253 1 0 Jul16 ? 00:02:23 httpssql
apache 29282 1 0 Jul16 ? 00:02:34 httpssql
apache 29296 1 0 Jul16 ? 00:01:52 httpssql
apache 29384 1 0 Jul16 ? 00:01:51 httpssql
apache 29462 1 0 Jul16 ? 00:02:59 httpssql
apache 29464 1 0 Jul16 ? 00:03:18 httpssql
apache 29466 1 0 Jul16 ? 00:02:14 httpssql
apache 29497 1 0 Jul16 ? 00:02:26 httpssql
apache 29541 1 0 Jul16 ? 00:02:40 httpssql
apache 29570 1 0 Jul16 ? 00:01:57 httpssql
apache 29582 1 0 Jul16 ? 00:02:03 httpssql
apache 29643 1 0 Jul16 ? 00:03:21 httpssql
apache 29718 1 0 Jul16 ? 00:02:50 httpssql
apache 29720 1 0 Jul16 ? 00:02:21 httpssql
apache 30164 1 0 Jul16 ? 00:02:43 httpssql
apache 30186 1 0 Jul16 ? 00:03:37 httpssql
apache 30277 1 0 Jul16 ? 00:02:56 httpssql
apache 30324 1 0 Jul16 ? 00:02:18 httpssql
apache 30390 1 0 Jul16 ? 00:02:15 httpssql
apache 30452 1 0 Jul16 ? 00:02:57 httpssql
apache 30495 1 0 Jul16 ? 00:02:59 httpssql
apache 30515 1 0 Jul16 ? 00:02:43 httpssql
apache 30541 1 0 Jul16 ? 00:03:03 httpssql
apache 30589 1 0 Jul16 ? 00:03:52 httpssql
apache 30675 1 0 Jul16 ? 00:02:30 httpssql
apache 30852 1 0 Jul16 ? 00:02:55 httpssql
apache 30941 1 0 Jul16 ? 00:01:35 httpssql
apache 31387 1 0 Jul16 ? 00:02:20 httpssql
apache 31576 1 0 Jul16 ? 00:02:06 httpssql
apache 31619 1 0 Jul16 ? 00:03:12 httpssql
apache 31650 1 0 Jul16 ? 00:02:38 httpssql
apache 31680 1 0 Jul16 ? 00:02:28 httpssql
apache 31691 1 0 Jul16 ? 00:02:45 httpssql
apache 31808 1 0 Jul16 ? 00:03:08 httpssql
apache 31876 1 0 Jul16 ? 00:02:55 httpssql
apache 32082 1 0 Jul16 ? 00:02:24 httpssql
apache 314 1 0 Jul16 ? 00:02:47 httpssql
apache 336 1 0 Jul16 ? 00:02:27 httpssql
apache 526 1 0 Jul16 ? 00:02:54 httpssql
apache 528 1 0 Jul16 ? 00:02:55 httpssql
apache 644 1 0 Jul16 ? 00:03:01 httpssql
apache 713 1 0 Jul16 ? 00:02:57 httpssql
apache 819 1 0 Jul16 ? 00:02:43 httpssql
apache 896 1 0 Jul16 ? 00:02:52 httpssql
apache 906 1 0 Jul16 ? 00:02:24 httpssql
apache 939 1 0 Jul16 ? 00:02:42 httpssql
apache 1046 1 0 Jul16 ? 00:01:50 httpssql
apache 1116 1 0 Jul16 ? 00:02:50 httpssql
apache 1126 1 0 Jul16 ? 00:02:16 httpssql
apache 1192 1 0 Jul16 ? 00:02:04 httpssql
apache 1623 1 0 Jul16 ? 00:02:29 httpssql
apache 1628 1 0 Jul16 ? 00:01:52 httpssql
apache 1656 1 0 Jul16 ? 00:01:57 httpssql
apache 1743 1 0 Jul16 ? 00:02:54 httpssql
apache 914 1 0 08:15 ? 00:00:11 httpssql
apache 947 1 0 08:15 ? 00:00:14 httpssql
apache 989 1 0 08:16 ? 00:00:07 httpssql
apache 1029 1 1 08:16 ? 00:00:16 httpssql
apache 1621 1 0 08:17 ? 00:00:09 httpssql
apache 4883 1 0 08:28 ? 00:00:00 httpssql
apache 5098 1 1 08:29 ? 00:00:08 httpssql
apache 5128 1 0 08:29 ? 00:00:00 httpssql
apache 5160 1 0 08:29 ? 00:00:00 httpssql
apache 5183 1 0 08:29 ? 00:00:00 httpssql
apache 5219 1 0 08:29 ? 00:00:00 httpssql
apache 5492 1 0 08:31 ? 00:00:04 httpssql
apache 5650 1 0 08:31 ? 00:00:00 httpssql
apache 6249 1 1 08:32 ? 00:00:08 httpssql


Any thoughts?

 

[wagnerch]

Thoughts are pretty much the same, did you follow the link I suggested and try downloading Coroner's Tookit and dumping the memory of the process?

Often there is evidence in the processes memory space about which domain name the attack is coming from.

Also try using lsof to see what files the processes have open. If you still need help and are not sure what to do then I would recommend hiring someone who has experience in these matters. You should also take inventory of every web application your clients are running and verify which ones have known vulnerabilities by checking the security websites.

Chad