High CPU Usage

[PaulJames]

Hey all, I'm fairly new to managing servers and I seem to have a problem, however, I do not seem to be able to work out what is causing it.

I keep receiving the alarm level changed email, and the status of the server is regularly "red".

I've doubled the size of the CPU power and increased the RAM on the server, but the alarms keep ringing.

I have read a few posts, and below I have pasted various results that I have got from the server, along with the alarm alert email.

I'm now completely and utterly stuck as to what I need to do.

Any help, would be appreciated

I'm happy to post more information as needed.

Thank you in advance

www.serveraddress.com: alarm level changed.

Server health parameter "CPU > Total usage" changed its status from "green" to "red".

top - 13:55:17 up 1 day, 23:28, 0 users, load average: 2.05, 2.04, 1.54
Tasks: 114 total, 2 running, 112 sleeping, 0 stopped, 0 zombie
Cpu(s): 27.8%us, 7.5%sy, 0.0%ni, 64.5%id, 0.2%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 3145728k total, 1731168k used, 1414560k free, 67360k buffers
Swap: 1959920k total, 0k used, 1959920k free, 976748k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
23640 userac 25 0 33348 5920 1176 S 102.0 0.2 23:02.91 perl
23891 userac 25 0 33344 5920 1180 R 100.0 0.2 19:21.40 perl
1 root 15 0 10352 752 628 S 0.0 0.0 0:01.87 init
2 root RT -5 0 0 0 S 0.0 0.0 0:00.21 migration/0
3 root 34 19 0 0 0 S 0.0 0.0 0:00.04 ksoftirqd/0
[cropped to allow posting]

# ls -l /proc/23640
total 0
dr-xr-xr-x 2 userac psacln 0 Jan 5 14:29 attr
-r-------- 1 userac psacln 0 Jan 5 14:29 auxv
-r--r--r-- 1 userac psacln 0 Jan 5 13:31 cmdline
-rw-r--r-- 1 userac psacln 0 Jan 5 14:29 coredump_filter
-r--r--r-- 1 userac psacln 0 Jan 5 14:29 cpuset
lrwxrwxrwx 1 userac psacln 0 Jan 5 14:29 cwd -> /var/tmp
-r-------- 1 userac psacln 0 Jan 5 14:29 environ
lrwxrwxrwx 1 userac psacln 0 Jan 5 13:59 exe -> /usr/bin/perl
dr-x------ 2 userac psacln 0 Jan 5 14:29 fd
-r--r--r-- 1 userac psacln 0 Jan 5 13:31 io
-r-------- 1 userac psacln 0 Jan 5 14:29 limits
-rw-r--r-- 1 userac psacln 0 Jan 5 14:29 loginuid
-r--r--r-- 1 userac psacln 0 Jan 5 14:29 maps
-rw------- 1 userac psacln 0 Jan 5 14:29 mem
-r--r--r-- 1 userac psacln 0 Jan 5 14:29 mounts
-r-------- 1 userac psacln 0 Jan 5 14:29 mountstats
-rw-r--r-- 1 userac psacln 0 Jan 5 14:29 oom_adj
-r--r--r-- 1 userac psacln 0 Jan 5 14:29 oom_score
lrwxrwxrwx 1 userac psacln 0 Jan 5 14:29 root -> /
-r--r--r-- 1 userac psacln 0 Jan 5 14:29 schedstat
-r--r--r-- 1 userac psacln 0 Jan 5 14:29 smaps
-r--r--r-- 1 userac psacln 0 Jan 5 13:31 stat
-r--r--r-- 1 userac psacln 0 Jan 5 13:55 statm
-r--r--r-- 1 userac psacln 0 Jan 5 13:32 status
dr-xr-xr-x 3 userac psacln 0 Jan 5 13:31 task
-r--r--r-- 1 userac psacln 0 Jan 5 14:29 wchan

# ls -l /proc/23891
total 0
dr-xr-xr-x 2 userac psacln 0 Jan 5 14:30 attr
-r-------- 1 userac psacln 0 Jan 5 14:30 auxv
-r--r--r-- 1 userac psacln 0 Jan 5 13:35 cmdline
-rw-r--r-- 1 userac psacln 0 Jan 5 14:30 coredump_filter
-r--r--r-- 1 userac psacln 0 Jan 5 14:30 cpuset
lrwxrwxrwx 1 userac psacln 0 Jan 5 14:30 cwd -> /var/tmp
-r-------- 1 userac psacln 0 Jan 5 14:30 environ
lrwxrwxrwx 1 userac psacln 0 Jan 5 13:59 exe -> /usr/bin/perl
dr-x------ 2 userac psacln 0 Jan 5 14:30 fd
-r--r--r-- 1 userac psacln 0 Jan 5 13:35 io
-r-------- 1 userac psacln 0 Jan 5 14:30 limits
-rw-r--r-- 1 userac psacln 0 Jan 5 14:30 loginuid
-r--r--r-- 1 userac psacln 0 Jan 5 14:30 maps
-rw------- 1 userac psacln 0 Jan 5 14:30 mem
-r--r--r-- 1 userac psacln 0 Jan 5 14:30 mounts
-r-------- 1 userac psacln 0 Jan 5 14:30 mountstats
-rw-r--r-- 1 userac psacln 0 Jan 5 14:30 oom_adj
-r--r--r-- 1 userac psacln 0 Jan 5 14:30 oom_score
lrwxrwxrwx 1 userac psacln 0 Jan 5 14:30 root -> /
-r--r--r-- 1 userac psacln 0 Jan 5 14:30 schedstat
-r--r--r-- 1 userac psacln 0 Jan 5 14:30 smaps
-r--r--r-- 1 userac psacln 0 Jan 5 13:35 stat
-r--r--r-- 1 userac psacln 0 Jan 5 13:55 statm
-r--r--r-- 1 userac psacln 0 Jan 5 13:37 status
dr-xr-xr-x 3 userac psacln 0 Jan 5 13:35 task
-r--r--r-- 1 userac psacln 0 Jan 5 14:30 wchan

# ps auxw | grep perl
root 29737 0.0 0.0 61148 768 pts/0 R+ 14:30 0:00 grep perl

# lsof -p 23640 |more
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
perl 23640 userac cwd DIR 253,1 58 172 /var/tmp
perl 23640 userac rtd DIR 202,1 4096 2 /
perl 23640 userac txt REG 253,0 13696 7411 /usr/bin/perl
perl 23640 userac mem REG 202,1 137256 310334 /lib64/ld-2.5.so
perl 23640 userac mem REG 253,0 1259888 12586129 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
perl 23640 userac mem REG 202,1 89800 310316 /lib64/libresolv-2.5.so
perl 23640 userac mem REG 202,1 111480 310331 /lib64/libnsl-2.5.so
perl 23640 userac mem REG 202,1 20424 310309 /lib64/libdl-2.5.so
perl 23640 userac mem REG 202,1 611880 310292 /lib64/libm-2.5.so
perl 23640 userac mem REG 202,1 45728 310317 /lib64/libcrypt-2.5.so
perl 23640 userac mem REG 202,1 15280 310350 /lib64/libutil-2.5.so
perl 23640 userac mem REG 202,1 142696 310291 /lib64/libpthread-2.5.so
perl 23640 userac mem REG 202,1 1712536 310298 /lib64/libc-2.5.so
perl 23640 userac mem REG 253,0 18080 4198489 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/IO/IO.so
perl 23640 userac mem REG 253,0 21424 12586731 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Socket/Socket.so
perl 23640 userac mem REG 202,1 53880 310340 /lib64/libnss_files-2.5.so
perl 23640 userac 0u unix 0xffff8800a679ef00 15659277 /var/run/mod_fcgid/sock/2180.975
perl 23640 userac 1w FIFO 0,6 15659594 pipe
perl 23640 userac 2w FIFO 0,6 15659594 pipe
perl 23640 userac 3u unix 0xffff88000004e600 15659407 /var/run/mod_fcgid/sock/2180.975
perl 23640 userac 4u IPv4 15659654 TCP www.serveraddress.com:40659->chi4.vm.bitvps.com:irdmi (ESTABLISHED)
perl 23640 userac 45r FIFO 0,6 5180 pipe
perl 23640 userac 48w FIFO 0,6 5181 pipe
perl 23640 userac 49w FIFO 0,6 15659188 pipe
perl 23640 userac 50w FIFO 0,6 15659242 pipe
perl 23640 userac 51w FIFO 0,6 15659189 pipe
perl 23640 userac 53w FIFO 0,6 15659190 pipe
perl 23640 userac 54w FIFO 0,6 15659243 pipe
perl 23640 userac 56w FIFO 0,6 15659244 pipe

# lsof -p 23891 |more
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
perl 23891 userac cwd DIR 253,1 58 172 /var/tmp
perl 23891 userac rtd DIR 202,1 4096 2 /
perl 23891 userac txt REG 253,0 13696 7411 /usr/bin/perl
perl 23891 userac mem REG 202,1 137256 310334 /lib64/ld-2.5.so
perl 23891 userac mem REG 253,0 1259888 12586129 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
perl 23891 userac mem REG 202,1 89800 310316 /lib64/libresolv-2.5.so
perl 23891 userac mem REG 202,1 111480 310331 /lib64/libnsl-2.5.so
perl 23891 userac mem REG 202,1 20424 310309 /lib64/libdl-2.5.so
perl 23891 userac mem REG 202,1 611880 310292 /lib64/libm-2.5.so
perl 23891 userac mem REG 202,1 45728 310317 /lib64/libcrypt-2.5.so
perl 23891 userac mem REG 202,1 15280 310350 /lib64/libutil-2.5.so
perl 23891 userac mem REG 202,1 142696 310291 /lib64/libpthread-2.5.so
perl 23891 userac mem REG 202,1 1712536 310298 /lib64/libc-2.5.so
perl 23891 userac mem REG 253,0 18080 4198489 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/IO/IO.so
perl 23891 userac mem REG 253,0 21424 12586731 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Socket/Socket.so
perl 23891 userac mem REG 202,1 53880 310340 /lib64/libnss_files-2.5.so
perl 23891 userac 0u unix 0xffff8800a679ef00 15659277 /var/run/mod_fcgid/sock/2180.975
perl 23891 userac 1w FIFO 0,6 15660479 pipe
perl 23891 userac 2w FIFO 0,6 15660479 pipe
perl 23891 userac 3w FIFO 0,6 15660479 pipe
perl 23891 userac 4u IPv4 15660485 TCP www.serveraddress.com:40673->chi4.vm.bitvps.com:irdmi (ESTABLISHED)
perl 23891 userac 45r FIFO 0,6 5180 pipe
perl 23891 userac 48w FIFO 0,6 5181 pipe
perl 23891 userac 49w FIFO 0,6 15659188 pipe
perl 23891 userac 50w FIFO 0,6 15659242 pipe
perl 23891 userac 51w FIFO 0,6 15659189 pipe
perl 23891 userac 53w FIFO 0,6 15659190 pipe
perl 23891 userac 54w FIFO 0,6 15659243 pipe
perl 23891 userac 56w FIFO 0,6 15659244 pipe

 

[MislavO]

I've answered you already yesterday and i got message "redirecting, this post will not be visable until moderator approved it......" .... so anyway.... i will write again.

It's good that you increased RAM/CPU power, but i think problem is somwhere else. When does your daily crons from Plesk run? Does this high CPU load happens only once per day on specific time?

Can you please tell me how many CPU cores do you have ? You can check that by pressing "1" when using command "top". By default Health Monitor in Plesk has kinda low values, so sometimes this critical errors can be false alarm.

Anyway, back to your problem. Try to run the following command:

# cd /var/ww/vhost
# find . -size +512M

(this will search for all files larger then 512MB)

Why should you run this? Once you run the command you will be looking specific for logs (ignore files in httpdocs structure, watch only statistics/logs/<LOG_NAME>). Maybe some logs are very big and daily cron can't handle that. Check this:
http://forum.parallels.com/showthread.php?t=264762
i had same problem.

If you find few GB logs and the problem you mentioned (high load) happened only once per day on specific time, this could be cause of your problem.

Altough if you don't find any big logs, you will've to investigate more deeply. Have you found anything specific in log files ?

 

[PaulJames]

Hey MislavO,

Thanks for replying to my post.

To confirm, the server is a 1and1 cloud server, with the following set-up:-
OS CentOS 5.5 (Final)
Panel version 11.0.9 Update #32
3GB RAM
100GB Space
The system is up-to-date; last checked at Jan 6, 2013 04:02 AM2 x Virtual Cores

The CPU has now been at 100% for 5 days, which is different than the usual error where it would only apply for a few hours.

I have checked the size of the logs, and they all appear to be normal sized (nothing over a few MB).

I'm quite concerned that my server has been hacked, or is in the process of being hacked

 

[MislavO]

Logs my friend, logs. First of all check mail queue. Then move to other logs, they will tell you something 1000%. There must be something in logs if CPU power is so high all the time.

Sign, if you have CPU on 100% for 5 days - should tell you that something really nasty is going on and you should've already contact your reseller where you broughted your server.

 

[PaulJames]

The website has worked fine, on lower CPU and RAM for a whole year, so unless I am being hacked I'm totally baffled.

We only have Web Server (Apache) and DNS Server (BIND) enabled and running in Services Management.

The rkhunter log shows similar to the guy on this post (most things "not found"):-
http://www.howtoforge.com/forums/showthread.php?t=44522

The only thing I can find in the logs, that looks out of place is repeated failed login attempts.

Extract from security log:-
Jan 8 15:14:07 s13588659 sshd[16869]: reverse mapping checking getaddrinfo for 101.18.173.59.broad.wh.hb.dynamic.163data.com.cn failed - POSSIBLE BREAK-IN ATTEMPT!
Jan 8 15:14:07 s13588659 sshd[16869]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=59.173.18.101 user=root
Jan 8 15:14:09 s13588659 sshd[16869]: Failed password for root from 59.173.18.101 port 56460 ssh2
Jan 8 15:14:10 s13588659 sshd[16884]: Received disconnect from 59.173.18.101: 11: Bye Bye
Jan 8 15:14:12 s13588659 sshd[17232]: Invalid user oracle from 59.173.18.101

I have a warning on the atMail error log (again, we don't use mail services):-
[Sun Jan 06 04:09:46 2013] [warn] RSA server certificate CommonName (CN) `Parallels Panel' does NOT match server name!?

sw-cp-server log shows:-
2013-01-08 09:37:43: (server.c.1543) server stopped by UID = 0 PID = 5232
2013-01-08 09:37:45: (log.c.166) server started
2013-01-08 09:42:43: (server.c.1543) server stopped by UID = 0 PID = 5933
2013-01-08 09:42:45: (log.c.166) server started

There are a few PHP errors, mainly connected to sqlite (we use mySQL)
PHP Warning: PHP Startup: Unable to load dynamic library '/usr/lib64/php/modules/sqlite.so' - /usr/lib64/php/modules/sqlite.so: cannot open shared object file: No such file or directory in Unknown on line 0

Using the firewall, I have blocked one of the frequent attackers IP addresses, this seems to have had some positive effect - however I now have increased "Load Average" (at intermittent times) and increased the number of running processes...