Tracking CPU Usage

[mdnitehk]

Hello,

I'm trying to figure out why my CPU Usage has jumped from about 1-2% up to about 8-10%. This has happened in the last week or so, and I'm not even sure where to look.

I rebooted my server last night, and everything was back where it has been, but now today again, it's pushing 9%. This really slows everything else down.

Thank you,
Jeff Homan

 

[rvdmeer]

what does TOP say?

[ronald@ptest ronald]# top

 

[mdnitehk]

I'm really not sure what you mean by TOP. Could you explain that more?

Thank you,
Jeff Homan

 

[rvdmeer]

execute top on the command line?

 

[Cranky]

10% CPU is nothing usually, it could be anything causing it and it's probably not a problem.

 

[mdnitehk]

Hello,

Here are the high CPU usage programs... there's alot of perl running. Is that typical or should I figure out where these scripts are and take care of them?

  PID USER     PRI  NI  SIZE  RSS SHARE STAT %CPU %MEM   TIME CPU COMMAND
 9865 apache    25   0   864  392   248 R    17.9  0.0 178:09   0 perl
16538 apache    25   0  2332 2184   516 R    17.9  0.4 131:21   0 perl
 2452 apache    25   0   868  396   248 R    17.7  0.0 357:47   0 perl
 9831 apache    25   0   872  396   248 R    16.5  0.0 179:49   0 perl
 1978 apache    25   0  2332  940   516 R    15.9  0.1 388:51   0 perl
10503 root      25   0   404  328    32 R    10.7  0.0 124:15   0 gzip

Thank you,
Jeff

 

[mdnitehk]

Also, I forgot to mention that the 10% causes a noticable decrease in performance. httpd seems fine, but email is pokey with usage that high... Is there a way to determine where the scripts are located when I see that the perl scripts are causing the majority of usage?

Thanks again,
Jeff

 

[rvdmeer]

Within TOP type: "cas1" (enter after it)
It will show you what the system is doing.
Or ofcourse: ps aux | grep perl

 

[lvalics]

Also if you hit on TOP ALT+w then will save the current settings.

 

[rvdmeer]

Great /me bows

 

[mdnitehk]

Detailed Info

Here's the details of the Perl scripts that are running:

  PID USER     PRI  NI  SIZE  RSS SHARE STAT %CPU %MEM   TIME CPU COMMAND
 9865 apache    25   0   864  392   248 R    19.8  0.0 342:27   0 perl udp.pl 216.108.184.7 6667 200
16538 apache    25   0  2328  992   508 R    19.8  0.1 295:37   0 [httpd]
 1978 apache    25   0  2328 1020   508 R    19.3  0.2 553:07   0 [httpd]
 9831 apache    25   0   872  396   248 R    15.3  0.0 344:06   0 perl udp.pl 69.14.241.252 6667 200
 2452 apache    25   0   868  396   248 R    14.8  0.0 522:04   0 perl udp.pl 200.40.119.72 80 6500
27575 apache    15   0  4032 3812   964 S     4.4  0.7  39:20   0 /usr/local/apache/bin/httpd -DSSL
27574 apache    15   0  4028 3972   964 S     3.9  0.7  39:06   0 /usr/local/apache/bin/httpd -DSSL

How can I find out what that script is doing, and if it should be running, and more importantly, how to stop it from running and hogging the CPU?

Thanks again, you've been a great help!

Jeff Homan

 

[rvdmeer]

I dont know if it rings a bell but port 6667 is an irc port... could be a webscript connecting to an external irc-server running on those 2 ip adresses?.

This could also mean your machine is rooted. (like infected) because in most cases when a server is infected a script will connect to an irc server so the owner is able to see what server is infected.

 

[rvdmeer]

And sadly enough.. I think what i thought is true

http://forums.serverbeach.com/archive/index.php/t-5194.html
http://www.linuxquestions.org/questions/archive/4/2004/12/3/261205

 

[mdnitehk]

I just found out that udp.pl is a perl script used for flooding... I'm still not sure how it got on my server, and how to delete it. I ran locate udp.pl, and I can't find it anywhere on the server. Any ideas there?

Thanks again,
Jeff

 

[bartje3]

Same troubles here, on 1 of my servers I have a CPU load between 2% - 45% (goes up and down) since my update to PSA 7.5.4 and php 4.4/mysql4.1.

When I do a top I see that apache is consuming a lot of it, but mysql has peaks between 80% - 98% CPU and 6.0% mem usage (during several seconds), something's wrong.

my.cnf was updated with the suggested settings (from Ivalics and others here in the forum), but with/without these settings...it stay's the same.

As I said, this happens only on one of my servers. To search after the problem I disabled Tomcat and Spamassassin, later I will disable the other services too to find the guilty one.

Bart

 

[rvdmeer]

I think its this script.

http://packetstormsecurity.nl/DoS/udp.pl

search for it on the customer home directories.

grep -ir "gr33ts: meth, etech, skrilla, datawar, fr3aky, etc" *

and block outgoing IRC traffic.

 

[rvdmeer]

P.s. Run this:

http://www.chkrootkit.org/

This will check and possible clear the script. But! i would reinstall the complete server...

 

[bartje3]

Thanks guys but:

- grep -ir "gr33ts: meth, etech, skrilla, datawar, fr3aky, etc" *

Returns nothing

- I use rkhunter (last version) and this gives me also nothing.

I installed mytop (similar to top) and I see peaks of MORE than 1000 qps (queries per second) ! How can I limit that ?

Thanks anyway, I will continue searching....

Bart

 

[mian]

the script is a udp flooder and they are using it to attack two irc servers and a web server. they obviously aren't very smart because they're trying to flood tcp ports using udp. because the scripts are running as user apache they probably used some kind of hole in your webserver or one of its scripts to execute it, an example of this would be using awstats 6.3 or lower. try grepping your apache log files for udp.pl and you might be able to find something like a wget command and which script on your server they exploited.

 

[rvdmeer]

mdnitehk, how are things going?