qmail remote queue fill-up, again

[echeng@]

Hello, guys. I've done many searches here and have found solutions to the standard qmail queue filling up issue, but my queue has just recently started filling up again, and I can't track the problem down.

A few months ago, my qmail queue started filling up quickly, and I did the following things to make it stop:

1. apply the 127.0.0.1 / 32 mail whitelist change

2. change all of my domains to "reject" unknown mail instead of "bouncing" it

3. cron a nightly, selective delete of messages with subject "failure notice" in the qmail queue


All has been smooth for several months, but mail started building up again, and my qmail queue last night was a staggering 345,000 messages! Looks like most of the messages are still failure notice messages, which is baffling.

Any ideas on how to track down the problem this time? I just confirmed again that all my domains are set to "reject" and not "bounce."

Also, what's an easy way to list qmail queue message information? I'm using qmHandle, which lets me selectively delete based on subject content, but when I use it to list huge queues, it just sits there. Maybe it takes a long time with so many queued messages...

Thanks for any help you might be able to give me...

 

[phoenixisp]

I have had no luck with the "Reject" function so I forward the unknown emails to an account I created with no mailbox or redirect. It sends the mail to a "black hole" so to speak.

 

[wagnerch]

Originally posted by echeng
All has been smooth for several months, but mail started building up again, and my qmail queue last night was a staggering 345,000 messages! Looks like most of the messages are still failure notice messages, which is baffling.

More than likely one of your domains was hacked and someone is using your local mail queue to spam. I would look for unusual jobs running as the apache user. Keep in mind that some of those jobs may appear to be "httpd", but you should pay particular attention to the "ppid".

There is a few worms running loose that exploit awstats and xmlrpc vulnerabilities. I ran across one guy who had 10~15K worth of e-mail's in his mail queue, and they were exploiting a vulnerability in xmlrpc. Installing modsecurity helps prevent alot of headaches.

 

[echeng@]

Thanks, guys, for the help. My system was indeed compromised, and seems to be working again.

Still, what I've done now is move to fastmail.fm mail, and have my DNS mx entries pipe mail directly there. I'm done with having server issues affect e-mail...

 

[echeng@]

thanks. I currently use qmHandle and qmailclear.sh.

I tried running qmHandle on 300,000 qmail messages, and ... well, it took some time.

 

[wagnerch]

Originally posted by echeng
Thanks, guys, for the help. My system was indeed compromised, and seems to be working again.

Still, what I've done now is move to fastmail.fm mail, and have my DNS mx entries pipe mail directly there. I'm done with having server issues affect e-mail...

Take a look at www.modsecurity.org, and then go to www.gotroot.com to download the modsecurity rules.

It takes a bit of tweaking, but once you tune the rules it will definately help. Naturally you have to keep on it, update the rules, etc. It is hard to control the domains if your doing web hosting, and there is so many vulnerable php scripts that there is no way to keep up.

Chad

 

[aheidl]

evil xml-rpc

i had a similar problem on my server which caused me plenty of sleepless nights. 250k+ mailes in a few hours which all seemed to be false bounces. I could delete those with qmHandle, but this was no way the solution to sit there and keep deleting them as you watch the remote que filling up.

apparently I found the source of all the evil to be the xml-rpc interface (php) as part of an postNuke (www.postnuke.org) installation. They did post a security warning about it and since I have disabled it I could stop the mess.

Maybe this is usefull to someone.

BR,

Alex

 

[wagnerch]

Re: evil xml-rpc

Originally posted by aheidl
apparently I found the source of all the evil to be the xml-rpc interface (php) as part of an postNuke (www.postnuke.org) installation. They did post a security warning about it and since I have disabled it I could stop the mess.

Yep, there is a Linux.Mare worm variant floating around exploiting XML-RPC and awstats. phpAdsNew is also vulnerable. There is atleast a dozen apps using XML-RPC.