Perl hung using 100% CPU every day

[RihadG]

Stock Debian 6.0.5 x86_64
Latest Plesk 11.x
This happens every day after a certain automated upgrade done by Plesk panel.
Nothing short of a kill -9 can stop these Perls.

top - 12:16:38 up 72 days, 18:35,  1 user,  load average: 5.43, 5.33, 5.14
Tasks: 226 total,   6 running, 220 sleeping,   0 stopped,   0 zombie
Cpu(s): 78.3%us, 21.6%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.1%si,  0.0%st
Mem:  16177148k total, 10932180k used,  5244968k free,   544664k buffers
Swap:  1951736k total,     2412k used,  1949324k free,  8799392k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                           
11351 www-data  20   0 32684 5404 1420 R   96  0.0 414:51.06 perl                                                                              
11264 www-data  20   0 32684 5456 1472 R   95  0.0 800:43.88 perl                                                                              
11356 www-data  20   0 32684 5400 1420 R   75  0.0 411:47.70 perl                                                                              
11336 www-data  20   0 32684 5400 1420 R   63  0.0 411:42.90 perl                                                                              
11191 www-data  20   0 32684 5428 1448 R   50  0.0 211:51.63 perl                                                                              
 1539 www-data  20   0  312m  46m 5060 S    5  0.3   0:00.18 apache2                                                                           
  483 www-data  20   0 62208  18m 1704 S    1  0.1   0:01.44 perl                                                                              
  487 www-data  20   0 60204  14m 1820 S    1  0.1   0:00.83 perl                                                                              
  537 www-data  20   0 62232  18m 1704 S    1  0.1   0:01.44 perl                                                                              
  543 www-data  20   0 59472  15m 1704 S    1  0.1   0:01.13 perl                                                                              
  548 www-data  20   0 60316  14m 1820 S    1  0.1   0:00.84 perl                                                                              
  565 www-data  20   0 59860  13m 1812 S    1  0.1   0:00.96 perl                                                                              
  664 www-data  20   0 59992  13m 1812 S    1  0.1   0:00.95 perl                                                                              
16331 bind      20   0  298m  68m 2532 S    1  0.4  42:33.21 named                                                                             
    1 root      20   0  8356  744  680 S    0  0.0   1:17.93 init                                                                              
  485 www-data  20   0 59420  15m 1704 S    0  0.1   0:01.13 perl                                                                              
  515 www-data  20   0 60176  14m 1796 S    0  0.1   0:00.88 perl                                                                              
  528 www-data  20   0 62232  18m 1704 S    0  0.1   0:01.43 perl                                                                              
  533 www-data  20   0 59472  15m 1704 S    0  0.1   0:01.14 perl                                                                              
  605 www-data  20   0 60152  14m 1796 S    0  0.1   0:00.89 perl                                                                              
  619 www-data  20   0 60132  14m 1796 S    0  0.1   0:00.89 perl                                                                              
  669 www-data  20   0 59984  13m 1812 S    0  0.1   0:00.95 perl                                                                              
 6956 www-data  20   0  292m  26m 4708 S    0  0.2   0:00.03 apache2                                                                           
 9089 www-data  20   0 61976  15m 1584 S    0  0.1   0:00.04 perl                                                                              
 9127 www-data  20   0 62596  16m 1584 S    0  0.1   0:00.04 perl                                                                              
 9444 www-data  20   0 63652  19m 1600 S    0  0.1   0:00.06 perl                                                                              
    2 root      20   0     0    0    0 S    0  0.0   0:00.00 kthreadd                                                                          
    3 root      RT   0     0    0    0 S    0  0.0   0:01.22 migration/0                                                                       
    4 root      20   0     0    0    0 S    0  0.0   3:34.04 ksoftirqd/0                                                                       
    5 root      RT   0     0    0    0 S    0  0.0   0:00.01 watchdog/0                                                                        
    6 root      RT   0     0    0    0 S    0  0.0   0:02.32 migration/1                                                                       
    7 root      20   0     0    0    0 S    0  0.0   0:35.44 ksoftirqd/1                                                                       
    8 root      RT   0     0    0    0 S    0  0.0   0:00.01 watchdog/1                                                                        
    9 root      RT   0     0    0    0 S    0  0.0   0:01.91 migration/2                                                                       
   10 root      20   0     0    0    0 S    0  0.0   0:33.25 ksoftirqd/2                                                                       
   11 root      RT   0     0    0    0 S    0  0.0   0:00.01 watchdog/2                                                                        
   12 root      RT   0     0    0    0 S    0  0.0   0:01.19 migration/3

 

[abdi]

http://forum.parallels.com/showthread.php?t=262791

 

[RihadG]

http://forum.parallels.com/showthread.php?t=262791

I don't think it's a DOS attack, traffic usage is normal. It's most probably a buggy update. 11.x is still beta.
But your tip might be useful as a last resort, thanks.

 

[Christian_S]

Try looking at open files and connections for the PIDs. It might point to some uwanted scripts.
eg. ls -l /proc/11351/fd

Regards
Christian

 

[RihadG]

Try looking at open files and connections for the PIDs. It might point to some uwanted scripts.
eg. ls -l /proc/11351/fd

Nice tip, but I just killed them a minute ago. I'll try that next time, thanks.

 

[RihadG]

Try looking at open files and connections for the PIDs. It might point to some uwanted scripts.
eg. ls -l /proc/11351/fd

Here's info for one of the offenders:
In top:
26382 www-data 20 0 39092 7664 1420 R 50 0.0 265:20.34 perl

# ll /proc/26382/fd

total 0
lr-x------ 1 www-data www-data 64 Sep 10 18:02 0 -> /dev/null
l-wx------ 1 www-data www-data 64 Sep 10 18:02 1 -> pipe:[2343584620]
lr-x------ 1 www-data www-data 64 Sep 10 18:02 146 -> pipe:[2343581874]
l-wx------ 1 www-data www-data 64 Sep 10 18:02 147 -> pipe:[2343581874]
lr-x------ 1 www-data www-data 64 Sep 10 18:02 148 -> pipe:[2343581875]
l-wx------ 1 www-data www-data 64 Sep 10 18:02 150 -> pipe:[2343581875]
l-wx------ 1 www-data www-data 64 Sep 10 18:02 151 -> /var/log/apache2/mod_jk.log
lrwx------ 1 www-data www-data 64 Sep 10 18:02 152 -> /var/log/apache2/jk-runtime-status.25516
lrwx------ 1 www-data www-data 64 Sep 10 18:02 153 -> /var/log/apache2/jk-runtime-status.25516.lock
lrwx------ 1 www-data www-data 64 Sep 10 18:02 156 -> socket:[2343584618]
l-wx------ 1 www-data www-data 64 Sep 10 18:02 2 -> /var/log/apache2/error.log
lrwx------ 1 www-data www-data 64 Sep 10 18:02 3 -> socket:[2343584635]

Nothing suspicious, if you ask me.

 

[RihadG]

It might be a virus or a bot...
According to top:
Tasks: 525 total, 1 running, 384 sleeping, 0 stopped, 140 zombie

# ps -ef|fgrep perl
www-data  3759 32191  0 21:37 ?        00:00:00 [perl] <defunct>
www-data 10877 32191  0 21:40 ?        00:00:00 [perl] <defunct>
www-data 20877     1  0 21:27 ?        00:00:00 perl anu.txt
www-data 20943     1  0 21:27 ?        00:00:00 perl anu.txt
www-data 20984     1  0 21:27 ?        00:00:00 perl anu.txt
www-data 21001     1  0 21:27 ?        00:00:00 perl anu.txt
www-data 21260 32191  0 21:44 ?        00:00:00 [perl] <defunct>
www-data 22933 32191  0 21:45 ?        00:00:00 [perl] <defunct>
www-data 23119 21001  0 21:28 ?        00:00:00 [perl] <defunct>
www-data 29144 32191  0 21:49 ?        00:00:00 [perl] <defunct>
www-data 29417 20984  0 22:37 ?        00:00:00 perl anu.txt
www-data 29418 20984  0 22:37 ?        00:00:00 [perl] <defunct>
www-data 29419 20984  0 22:37 ?        00:00:00 [perl] <defunct>
www-data 29420 20984  0 22:37 ?        00:00:00 [perl] <defunct>
www-data 29421 20984  0 22:37 ?        00:00:00 [perl] <defunct>
www-data 32189 21001  0 21:35 ?        00:00:00 perl anu.txt
www-data 32190 21001  0 21:35 ?        00:00:00 perl anu.txt
www-data 32191 21001  0 21:35 ?        00:00:00 perl anu.txt
www-data 32192 32190  0 21:35 ?        00:00:00 perl anu.txt
www-data 32323 32189  0 22:38 ?        00:00:00 perl anu.txt
www-data 32324 32189  0 22:38 ?        00:00:00 [perl] <defunct>
www-data 32325 32189  0 22:38 ?        00:00:00 [perl] <defunct>
www-data 32326 32189  0 22:38 ?        00:00:00 [perl] <defunct>
www-data 32327 32189  0 22:38 ?        00:00:00 [perl] <defunct>
www-data 32328 32189  0 22:38 ?        00:00:00 [perl] <defunct>
www-data 32329 32189  0 22:38 ?        00:00:00 perl anu.txt
www-data 32330 32189  0 22:38 ?        00:00:00 perl anu.txt
www-data 32331 32189  0 22:38 ?        00:00:00 [perl] <defunct>
www-data 32332 32189  0 22:38 ?        00:00:00 perl anu.txt
www-data 32333 32189  0 22:38 ?        00:00:00 [perl] <defunct>
www-data 32337 20943  0 22:38 ?        00:00:00 perl anu.txt
www-data 32338 20943  0 22:38 ?        00:00:00 [perl] <defunct>
www-data 32339 20943  0 22:38 ?        00:00:00 [perl] <defunct>
www-data 32340 20943  0 22:38 ?        00:00:00 [perl] <defunct>
www-data 32341 20943  0 22:38 ?        00:00:00 [perl] <defunct>
www-data 32342 20943  0 22:38 ?        00:00:00 [perl] <defunct>
www-data 32343 20943  0 22:38 ?        00:00:00 perl anu.txt
www-data 32344 20943  0 22:38 ?        00:00:00 perl anu.txt
www-data 32345 20943  0 22:38 ?        00:00:00 [perl] <defunct>
www-data 32346 20943  0 22:38 ?        00:00:00 perl anu.txt
www-data 32347 20943  0 22:38 ?        00:00:00 [perl] <defunct>
www-data 32375 32415  0 22:38 ?        00:00:00 perl anu.txt
www-data 32376 32415  0 22:38 ?        00:00:00 [perl] <defunct>
www-data 32377 32415  0 22:38 ?        00:00:00 [perl] <defunct>
www-data 32378 32415  0 22:38 ?        00:00:00 [perl] <defunct>
www-data 32379 32415  0 22:38 ?        00:00:00 [perl] <defunct>
www-data 32380 32415  0 22:38 ?        00:00:00 [perl] <defunct>
www-data 32381 32415  0 22:38 ?        00:00:00 perl anu.txt
www-data 32382 32415  0 22:38 ?        00:00:00 perl anu.txt
www-data 32383 32415  0 22:38 ?        00:00:00 [perl] <defunct>
www-data 32384 32415  0 22:38 ?        00:00:00 perl anu.txt
www-data 32385 32415  0 22:38 ?        00:00:00 [perl] <defunct>
www-data 32389 20877  0 22:38 ?        00:00:00 perl anu.txt
www-data 32390 20877  0 22:38 ?        00:00:00 [perl] <defunct>
www-data 32391 20877  0 22:38 ?        00:00:00 [perl] <defunct>
www-data 32392 20877  0 22:38 ?        00:00:00 [perl] <defunct>
www-data 32393 20877  0 22:38 ?        00:00:00 [perl] <defunct>
www-data 32394 20877  0 22:38 ?        00:00:00 [perl] <defunct>
www-data 32395 20877  0 22:38 ?        00:00:00 perl anu.txt
www-data 32396 20877  0 22:38 ?        00:00:00 perl anu.txt
www-data 32397 20877  0 22:38 ?        00:00:00 [perl] <defunct>
www-data 32398 20877  0 22:38 ?        00:00:00 perl anu.txt
www-data 32399 20877  0 22:38 ?        00:00:00 [perl] <defunct>
www-data 32415     1  0 21:10 ?        00:00:00 perl anu.txt
www-data 32420 29417  0 22:38 ?        00:00:00 perl anu.txt
www-data 32421 29417  0 22:38 ?        00:00:00 perl anu.txt
www-data 32422 29417  0 22:38 ?        00:00:00 perl anu.txt
www-data 32423 29417  0 22:38 ?        00:00:00 perl anu.txt
www-data 32424 29417  0 22:38 ?        00:00:00 perl anu.txt
www-data 32425 29417  0 22:38 ?        00:00:00 perl anu.txt
www-data 32426 29417  0 22:38 ?        00:00:00 perl anu.txt
www-data 32427 29417  0 22:38 ?        00:00:00 perl anu.txt

There's no anu.txt anywhere according to find.