QMail, Spam, Unknown Injection Source, Angry Clients, Oh My!

[criticman]

Alright, somehow a new injection attempt was successful by a spammer...but no clue how!

At first it looked like through Perl...and it may have been. I checked /tmp, nothing of interest, but I cleared it anyway.

I used

qmHandle -Senviou

to delete the spam (950,000+) on Wednesday.

Well, things seemed to be going alright. I had killed the Perl process, did a readlink on it prior to killing it and it turned up nothing.

I had to rebuild the queue, finally got mail working, and all was good....

Then today I realize a cron report hadn't arrived, so I checked the queue and found 750,000+ messages. Same subject as the previous 950,000 ones.

So, I checked processes again. Nothing! I am deleting the spam again (with Qmail disabled).

So, let's look at this from two angles.

1) Let's assume it was an injection again. In case Perl was the culprit, I disabled it on all of my domains as I don't use it. Any other ways to lock it down?

2) Let's assume that it was not an injection and that someone managed to relay...how? I believe all settings are made to prevent this from happening. They were being sent as [email protected], but I do not even have the mail server enabled for mydomain.com as I am using GMail for Domains Beta for mydomain.com. How were they able to use this address to send it? Would that point to an injection and not a relay?

How can I set up a firewall to block IP ranges from countries my clients do not do busines in? I know the Plesk firewall module cannot do this, so does anyone have a link to a tutorial on how to block country IP ranges using ipchains/iptables?

Also, I would like to try to see if the source of the messages is still on the server. Since the subject had "enviou", I am running

grep -r enviou /*

from the base / dir. Will that work or is there a faster command?

 

[faris]

This sounds like a horrible situation -- you have my sympathies.

I can't help you with very much on this, but I can suggest a way to block emails from various countries. All you have to do is use the MAPS function in Plesk and use one of the RBLs that blocks based on country. I can't point you in a specific direction, but I know they exists - I came across several not so long ago.

Faris.

 

[criticman]

Does using MAPS block both outgoing and incoming mail? Or will it only work to block incoming mail from known spam locations?

 

[faris]

Not as such. It does work on any incoming smtp, in that if you had a user in China and you had blocked chinese IPs using an RBL, they would not be able to connect and send email using authenticated smtp, but it won't stop a script running on the server sending to a Chinese IP, nor will it stop a non-chinese IP sending to a Chinese IP.

Faris.

 

[criticman]

However, a properly configured firewall will prevent incoming AND outgoing traffic to foreign IP addresses. This would include someone from a foreign IP trying to use HTTP to hijack or inject and if they suceeded from a non-blocked IP (a slave machine in their control) and the script tried to open a connection to their foreign IP, it should be blocked.

So, anyone want to chime in about proper steps to block foreign IP ranges?

 

[atomicturtle]

qmail-scanner will scan both in and outbound messages, if you set it to delete, it should help stem the tide as it were, while you're looking for the culprit.

Other logs you'll want to look through:

/var/log/httpd/suexec_log <- perl will be logged here
/var/log/maillog and /usr/local/psa/var/log/maillog
/var/log/messages <- smtp_auth logs here, with usernames. I see spammers using this more and more all the time.
/var/log/secure <- SMTP connections are logged here
/var/spool/qmailscan/qmail-queue.log <- qmail-scanner debug log


Make sure you've turned poplocking off, if you've got users behind proxies (AOL for example), its going to allow all of AOL to relay through your box.

 

[wagnerch]

I would post the results of this:

ps -fu apache

If you notice anything that has a PPID of 1, then odds are one of your sites was hacked and someone is injecting spam directly into the local queue. I have seen it many times.

 

[criticman]

So I had things "fixed" for a bit, then it happened again:

ps -fu apache
UID PID PPID C STIME TTY TIME CMD
apache 29539 25104 0 May28 ? 00:00:31 /usr/sbin/httpd
apache 29540 25104 0 May28 ? 00:00:28 /usr/sbin/httpd
apache 29541 25104 0 May28 ? 00:00:28 /usr/sbin/httpd
apache 29543 25104 0 May28 ? 00:00:33 /usr/sbin/httpd
apache 29544 25104 0 May28 ? 00:00:29 /usr/sbin/httpd
apache 29545 25104 0 May28 ? 00:00:29 /usr/sbin/httpd
apache 29546 25104 0 May28 ? 00:00:29 /usr/sbin/httpd
apache 26463 25104 0 May28 ? 00:00:32 /usr/sbin/httpd
apache 29110 25104 0 May28 ? 00:00:27 /usr/sbin/httpd
apache 30435 25104 0 May28 ? 00:00:26 /usr/sbin/httpd
apache 11682 1 0 May28 ? 00:00:00 /bin/sh
apache 25450 25104 0 May29 ? 00:00:20 /usr/sbin/httpd
apache 26798 25104 0 May29 ? 00:00:20 /usr/sbin/httpd
apache 29316 25104 0 May29 ? 00:00:17 /usr/sbin/httpd
apache 30529 25104 0 May29 ? 00:00:15 /usr/sbin/httpd
apache 30531 25104 0 May29 ? 00:00:18 /usr/sbin/httpd
apache 30537 25104 0 May29 ? 00:00:17 /usr/sbin/httpd
apache 30540 25104 0 May29 ? 00:00:15 /usr/sbin/httpd
apache 30546 25104 0 May29 ? 00:00:18 /usr/sbin/httpd
apache 30549 25104 0 May29 ? 00:00:13 /usr/sbin/httpd
apache 24691 25104 0 May30 ? 00:00:12 /usr/sbin/httpd

 

[criticman]

~]# readlink /proc/11682/exe
/bin/bash

 

[wagnerch]

Yeah, that is screwy. I would run "ps fuxwa" and locate pid 11682 and look at any children processes. Odds are if the "bash" isn't doing something then they managed to obtain root priviledge or another account.

 

[MarcusInMd]

I have (had) the following on our server running...

UID PID PPID C STIME TTY TIME CMD
apache 2400 1 0 Apr30 ? 00:00:00 ./ulib
apache 19119 1 0 Apr30 ? 00:00:00 ./ulib
apache 19751 28933 0 May02 ? 00:01:15 /usr/sbin/httpd
apache 4506 1 0 May03 ? 00:00:00 ./bt


Is that something to be concerned with? We were not having any kinds of issues that I am aware of but I decided to see what was going on when I came accross this thread.

 

[wagnerch]

Yes, definately. No process should run as the apache user with a PPID (parent process id) of 1. PPID of 1 indicates a background job, and that is just not the behavior of apache. Apache, even when running CGIs, will have the child processes attached to an httpd process. The only httpd process that runs with PPID of 1 is owned by root (normally).

I would say those 3 processes are highly suspicious, and wouldn't be suprised if something isn't going on. There really is no reason for a backgrounded (detached) process to be running that was executed by apache.

 

[MarcusInMd]

Thanks for the quick reply.

Where should I go from here?

I have the server locked down as best as I can do it. I routinly run chkroot and rkhunter and have never come across anything unusual.

I killed those processes...Should I watch and see if they start up again and if they do, how can I track it down?

 

[wagnerch]

You may want to investigate using modsecurity, but I must warn you that there are some clear positives and negatives with modsecurity:

1. it is rule based and uses patterns, which ultimately means it cannot defeat all application attacks.
2. some rules are clearly flawed and affect legit applications.


What you need to do is assess your domain accounts and identify what versions of what applications they are running. Then you need to validate whether those applications have known security holes. Some very popular web apps are known to have security holes, such as phpBB, Mambo, PostNuke, phpAds, etc.

If the applications pop up again, then I highly recommend dumping the memory space of the application using The Coroner's Toolkit. Search on the forums here for the Coroner's Toolkit, there is a patch file to build on Linux 2.6. "pcat" is the process that comes with TCT that allows you to dump a processes memory space, from there I usually look at the dump using strings and try to lift some information from it. Another thing you want to do is identify *WHEN* the process started, that is the "START" column in ps. Once you identify when it started you should look in ALL of your domain's access logs and try to correlate an HTTP request that occurred either at the same time or a few seconds earlier.

 

[nbdev]

On our case such mail is in almost all cases because of a user programming a php-script without sufficient protection which enables spammers to use the mail()-function.

Investigating the message can sometimes lead to the domain where the cause can be found.

When no sender-address is set the mail-address [email protected] is automatically used.

Hope this helps

 

[MarcusInMd]

I think I figured out what happened. I will try to explain it later, I have got a service call to go on in a few minutes.

But in short, I believe the problem started with a vulnerable application (Invision Power Board)

I patched it shortly after I was notified that a problem had occured. Someone had injected code into a post and gained access to the forum software. I have to look further into what it was the script did. I will post more later.

 

[alexhubner]

Relay for local domains?

I don't know if it's the case, but I'm VERY worried about a potential flaw in Plesk's Qmail that allows a spammer to send mail to local domains without providing credentials (open relay). I really can't understand why this is possible.

More details in:

http://forums.ev1servers.net/showthread.php?t=62149

 

[wagnerch]

Re: Relay for local domains?

Originally posted by alexhubner
I don't know if it's the case, but I'm VERY worried about a potential flaw in Plesk's Qmail that allows a spammer to send mail to local domains without providing credentials (open relay). I really can't understand why this is possible.

Uhh, that is the way e-mail is supposed to work. How would you expect me to send an e-mail to domainxyz.com if I couldn't deliver it without credentials?

The server will ONLY require credentials if the "TO" is not in the server's "rcpthosts", otherwise you simply would receive no mail -- EVER! Relaying is when the "origin" is not in "locals" and the destination is not in "rcpthosts", then authentication is required.

 

[alexhubner]

I understand that this is a expected behaviour, but in Plesk this behaviour allows a malicious user to send messages to your domain as a legitimate user. It's just a matter of (1) guess a valid e-mail address on the domain (ie. [email protected]) and (2) use the domain's SMTP server (ie. smtp.yourdomain.com) to send messages (virus, anything) to other emails/users in that same domain.

You can do it many ways, one is using a simple email client such as Outlook or Thunderbird, other is using a script or even creating a worm to perform that. I believe this is not a good thing, right?

I'm not a MTA/SMTP expert, but I know that there's a way to force users to authenticate before sending messages throught SMTP servers even if the message is addressed to a domain allowed to relay in the rcpthosts file. I just don't know how, so I'll try to exemplify with real examples (both using Qmail, but one from a flawed (imho) Plesk Qmail install):

DOMAIN: iventure.com.br (hosted in a Plesk box)
DOMAIN: hubner.org.br (hosted in a Linux/Qmail box)

The SMTP servers are: smtp.iventure.com.br and smtp.hubner.org.br

If you try to send message to a local domain in the "iventure.com.br" server, you'll *not* be asked for credentials. See below:

TELNETING SMTP.IVENTURE.COM.BR 25:

220 mail7.worldispnetwork.com ESMTP
helo
250 mail7.worldispnetwork.com
mail from:[email protected]
250 ok
rcpt to:[email protected]
250 ok
data
354 go ahead
subject=teste
asas
.
250 ok 1149023436 qp 11545
quit
221 adt2.amazonia.org.br
Connection closed by foreign host.

Now, this is impossible with a secure SMTP server such as SMTP.HUBNER.ORG.BR. See below (and test by yourself):

TELNETING SMTP.HUBNER.ORG.BR 25:

220 smtp.hubner.org.br ESMTP
helo
250 smtp.hubner.org.br
mail from:[email protected]
250 ok
rcpt to:[email protected]
553 THIS SERVER IS TO BE USED WITH AUTHENTICATION (#5.7.1)
Connection to host lost.

Of course I'm not talking about regular mail delivery from one SMTP system to another. I'm talking about sending messages (creating/originating messages) via Plesk, which is flawed imho.

Currently I'm receiving virus from users who are not legitimate users, but can bypass any protection such as SPF and authentication since they are sending messages FROM our domain and WITHIN our domain whitout being required for a credential. That's a serious flaw if you ask me, and I'm trying to mitigate it somehow.

 

[wagnerch]

Alex, I don't know if you have ever looked at Qmail. It wouldn't suprise me if Plesk is lacking some feature of Qmail that you have from another packager. Qmail, IMO, is a poorly maintained package. Any package that has 5 billion patches on their website is an indicator of how it is maintained, those patches should be integrated in with the core package and released. Qmail is touted as one of the most secure MTAs, but I doubt that extends to all of these patches that packagers integrate from various sources.

It might be a good idea to start a new thread, and see if anyone with Qmail expertise in this area can help. I have alot of experience with MTA's, but Qmail is relatively new to me.

It appears what your asking for is authentication to be forced if the FROM is in your rcpthosts, regardless of whether it is being delivered TO something in your rcpthosts. Most MTA's allow this behavior of forging e-mail, it is difficult to control or prohibit because there is legitimate cases where this behavior is necessary. It is not considered relaying, atleast not in my experience. Relaying is typically defined as the RCPT and TO are outside of my "rcpthosts" or "locals".