qmail forking and bombing system with messages

[theladyboo]

I first noticed when I couldn't ftp because the system was running out of memory. So I went in with ssh and poked around.

If I start qmail with plesk it dies. If I start it from the xinetd starup file from the command line it works fine except I don't get any mail.

I checked the apache logs for any scripts and nothing was unusual. I found a lot of strange things where people were trying, but getting 404 errors.

I tried turning off all the web servers and then turning on qmail through plesk, but the hits are still there according to netstat and the server is still brought down to a crawl. This was after I killed all the qmail processes and made sure it wasn't running.

I wanted to update qmail, but then in the process i had to go hunt down a patch tool because the server didn't have one and then I thought about it and wondered if please would even be compatible with the upgrade so I thought I'd ask here first.

An example would be:

65.116.31.17 - - [07/Apr/2007:11:28:19 -0500] "GET http://www.microsoft.com/ HTTP/1.0" 404 12826 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)"
65.116.31.17 - - [07/Apr/2007:11:28:20 -0500] "POST http://lti-mail01.ltinetworks.com:25/ HTTP/1.0" 404 12826 "-" "-"
24.172.195.8 - - [07/Apr/2007:11:50:59 -0500] "GET http://www.microsoft.com/ HTTP/1.0" 404 12826 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)"
24.172.195.8 - - [07/Apr/2007:11:51:00 -0500] "POST http://lti-mail01.ltinetworks.com:25/ HTTP/1.0" 404 12826 "-" "-"
207.151.97.218 - - [07/Apr/2007:11:52:55 -0500] "POST http://lti-mail01.ltinetworks.com:25/ HTTP/1.0" 404 12826 "-" "-"
207.151.97.218 - - [07/Apr/2007:11:52:56 -0500] "GET http://www.microsoft.com/ HTTP/1.0" 404 12826 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)"
206.165.199.101 - - [07/Apr/2007:12:07:37 -0500] "GET / HTTP/1.0" 200 14480 "-" "-"
206.165.199.101 - - [07/Apr/2007:12:07:54 -0500] "GET / HTTP/1.0" 200 14480 "-" "-"

and then I'll see seomthing like this which makes no sense because

59.10.167.48 - - [16/Sep/2006:14:28:42 -0700] "GET / HTTP/1.0" 200 4854 "http://www.openfos.com/supply/ALLIED-STEEL-CONSTRUCTION-CO-L-34270/" "Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0)"


Netstat:

netstat had a ton of processes similar to
tcp 0 1 ip-xxx-xx-xxx-xxx.ip.:33430 mx1.csbc.com:auth SYN_SENT
tcp 0 0 ip-xxx-xx-xxx-xxx.ip.s:smtp mx1.silcon.com:51502 ESTABLISHED
tcp 0 0 ip-xxx-xx-xxx-xxx.ip.s:smtp kos.kasamba.com:13020 TIME_WAIT

my qmail pids are similar to this (and there are a ton of them)

3778 ? Ss 0:00 /var/qmail/bin/qmail-smtpd /var/qmail/bin/smtp_auth /var/qmail/bin/true /var/qmail/bin/cmd5checkpw /var/qmail/bin/true
7395 ? Ss 0:00 tcp-env /var/qmail/bin/relaylock /var/qmail/bin/qmail-smtpd /var/qmail/bin/smtp_auth /var/qmail/bin/true /var/qmail/bin/cmd5checkpw /var/qmail/bin/true
422 ? Ss 0:00 /var/qmail/bin/qmail-smtpd /var/qmail/bin/smtp_auth /var/qmail/bin/true /var/qmail/bin/cmd5checkpw /var/qmail/bin/true
7177 ? S 0:00 qmail-send
7179 ? S 0:00 splogger qmail
7180 ? Z 0:00 [qmail-lspawn] <defunct>
7182 ? Z 0:00 [qmail-rspawn] <defunct>
7186 ? Z 0:00 [qmail-clean] <defunct>
(the above happened when I renamed the bin directory to stop)
18251 ? Ss 0:00 xinetd -stayalive -pidfile /var/run/xinetd.pid
18431 ? S 0:00 plugins/chkrcptto

I don't see any unusual processes active

If I start qmail using /etc/init.d/qmail start then the processes do not startup yet mail doesn't work.

If I start qmail using /etc/init.d/qmail start then the problem does not occur.

I've looked for php scripts in my apache error and access logs and nothing irregular has appeared.

When I turned on qmail through plesk I would get these in /usr/local/psa/var/log/maillog

Apr 7 11:40:36 ip-216-69-172-172 qmail: 1175964036.353178 alert: unable to opendir todo/0, sleeping...
Apr 7 11:40:59 ip-216-69-172-172 relaylock: /var/qmail/bin/relaylock: mail from 71.16.41.208:18242 (mailgw.channelblade.com)
Apr 7 11:40:59 ip-216-69-172-172 relaylock: /var/qmail/bin/relaylock: mail from 69.10.230.29:57883 (webmail6.mhsmail.onx.com)
Apr 7 11:40:59 ip-216-69-172-172 relaylock: /var/qmail/bin/relaylock: mail from 66.148.195.190:1364 (66.148.195.190.nw.nuvox.net)
Apr 7 11:40:59 ip-216-69-172-172 relaylock: /var/qmail/bin/relaylock: mail from 141.20.1.74:53743 (suncom4.cms.hu-berlin.de)
Apr 7 11:40:59 ip-216-69-172-172 relaylock: /var/qmail/bin/relaylock: mail from 67.102.68.194:7980 (h-67-102-68-194.snfccasy.covad.net)
Apr 7 11:40:59 ip-216-69-172-172 relaylock: /var/qmail/bin/relaylock: mail from 86.64.52.71:4914 (71.52.64-86.rev.gaoland.net)
Apr 7 11:40:59 ip-216-69-172-172 relaylock: /var/qmail/bin/relaylock: mail from 83.206.128.2:41465 (ns3.cnce.caisse-epargne.fr)
Apr 7 11:40:59 ip-216-69-172-172 relaylock: /var/qmail/bin/relaylock: mail from 66.147.88.49:49436 (nsc66.147.88-49.newsouth.net)
Apr 7 11:40:59 ip-216-69-172-172 relaylock: /var/qmail/bin/relaylock: mail from 63.139.215.138:33132 (mail.dnlukems.com)
Apr 7 11:40:59 ip-216-69-172-172 relaylock: /var/qmail/bin/relaylock: mail from 85.33.97.170:12055 (host170-97-static.33-85-b.business.telecomitalia.it)
Apr 7 11:40:59 ip-216-69-172-172 relaylock: /var/qmail/bin/relaylock: mail from 64.65.207.202:29635 (rochester.wardsupply.com)
Apr 7 11:40:59 ip-216-69-172-172 relaylock: /var/qmail/bin/relaylock: mail from 81.223.16.242:18879 (mail.ic-vienna.at)
Apr 7 11:41:00 ip-216-69-172-172 relaylock: /var/qmail/bin/relaylock: mail from 58.185.11.132:7781 (not defined)
Apr 7 11:41:00 ip-216-69-172-172 relaylock: /var/qmail/bin/relaylock: mail from 192.116.223.134:6172 (owa.eyron.com)

The dates on the qmail files are normal, although that can be changed.

There were some log files missing in the system log.

Any ideas or information I've missed to tell you

My other option is reprovisioning the server and that will be a nightmare because I have a lot going on and ftp isn't the most reliable transfer for large data. If I miss something I'll be screwed.

Thanks for your help.

Rebecca

 

[theladyboo]

I'm just going to try and reinstall the rpm. Unfortunately godaddy didn't put the rpm in my root's home. Does anyone know where godaddy stores the Plesk RPMs for emergencies?

Thanks

 

[theladyboo]

Any ideas at all?

I did a clean install of qmail and that didn't solve the problem.

I did a flow tail of the messages log for all the web servers and turned it on and nothing came up.

I've checked the tmp and temp directories as well as others for anything unusual and nothing.

I turned off apache and started the email and it still happens.

 

[atomicturtle]

Ouch, sounds like someones compromised the system then. Do you have good backups?

 

[theladyboo]

Yes, there's just so much data it's going to be a pain. Thanks.

 

[atomicturtle]

Id run through some of the forensics tools first, like rkhunter and chkrootkit. See if there are any unsual processes being run as the apache user. If there are, then you'll want to see if you can identify the vector of attack used by the badguys before you rebuild the system. Otherwise you'll just repeat this whole affair again once you're done.

 

[theladyboo]

Hi,

I did run a couple of the tools and a few little tricks like echo'ing the results and it didn't find a darn thing. I did find a few chroot directories that I removed, but I'm not sure if they were plesk related or not.

Unfortunately it is still happening and it looks like the problem might be with a few of the linux lib files. Since I don't have the original rpms that the server has and I'm not sure if I have to rebuild the kernel at that point, I don't know if it would be easier to just redo the server.

 

[theladyboo]

O M G

So I backed up all my vhosts and databases then asked the company to reprovision the server. They claim they did

so

the problem is still happening

even worse

they said it was reprovisioned which to me meant everything was wiped and put on, right? Some things have today's date but the rest has the original install date as do the log files. Plesk won't boot either.

I'm just stressing

 

[theladyboo]

Okay, so I backed up my stuff and reprovisioned the server (like 3 or 4 times!) and it never got rid of the problem and then the #$%@!@#$!% hosting company shut down my server because they said I had a service violation.

Um...I don't think so!

I paid for a full year and then this started happening. these guys had better refund my money and I mean now. (no, not plesk, the webhosting company).