Recording all outgoing mail to stop spam

[madcat]

Hello. I am trying to stop outgoing spam from coming from our servers. I believe our spam is due to the fact that users choose stupid passwords, and spammers use brute force scripts to guess these passwords, and then use other scripts to connect to the webmail program to send spam. I have searched this forum and found that I am definitely not the only one suffering from this problem.

What I would like to do, as a solution, is setup a log file that records _ALL_ outgoing email coming from the server, regardless of what method is used to send the mail. Right now, I have replaced /usr/bin/sendmail with a script that writes to a log file all the info about the message (including the value of $CWD), which tells me what directory /usr/bin/sendmail was called from, which tells me who on my server actually sent the spam (or who's account was compromised to send the spam). The problem is that many of the programs that send mail do so by directly connecting to port 25 on my server, bypassing the sendmail binary.

I have also tried searching through the log file located at /usr/local/psa/var/log/maillog . The problem with that one is that it records the address the spammer provides as the "from" address, which is almost always fake. Even if it is a real address, it doesn't tell me which account _ON_MY_SERVER_ is responsible for sending the message.

What I would love to have is a plain text log file, with three columns, with this format:

[Date-Stamp] [Sending Account] [Subject-Line]

Where [Sending Account] is the FQDN mailname used to log into webmail, or authenticate to SMTP. In the case where a PHP or Perl script is being used to send spam, the [Sending Account] would instead be the domain name. I would then read this log file once per day, and would be able to tell how the spam is coming out of my server.

For most cases, reading the [Subject-Line] would tell me whether or not the message is spam. I could even sort the message by [Sending Account] and simply count the number of messages sent. A large number of messages indicates a possible compromised account (or a blatant spammer singing up). Is there even any way to determine how many messages a given domain has sent via the Plesk interface? I haven't found one, so far.

Thanks.

 

[Amin Taheri]

Try these

make sure open relay is off
http://kb.swsoft.com/en/1394

Create wrapper for sendmail to log all mail sending from web scripts
http://kb.swsoft.com/en/1711

 

[madcat]

Thanks for the reply, HostingGuy. But I've already implemented http://kb.swsoft.com/en/1711 and made sure the server isn't an open relay. What I'm hoping to do is similar to http://kb.swsoft.com/en/1711 , but instead it records connections to port 25 on localhost.

 

[madcat]

I'm getting somewhere ... I found http://rackerhacker.com/2007/02/10/finding-compromised-mail-accounts-in-plesk

However, the smtp_auth messages it talks about do not show up in /var/log/messages ... anybody know why? The server setting are Relaying -> Authorization is Required -> SMTP

 

[madcat]

The issue might be that STARTTLS is being used so the login username is encrypted ... anybody know how to capture the login credentials being used for TLS into a log file?

 

[Amin Taheri]

maybe in /var/log/secure ?

 

[madcat]

Yup ... I'm finding entries in /var/log/secure with IP addresses that corespond to the connections, but not the username they used to authenticate ... and that's the problem ... sure, I could start banning all these IP's but other spammers will just start using it ... I need to figure out which account is being used, and change it's password.

As much as I like the idea of encrypted communications, in this case it is a burden since I am not able to see the username in any logs. However, it must decrypt the username & password at some point in time, and it would be nice if this were logged somewhere, somehow.

Do I have to edit config files? Recompile qmail? If recompiling is necessary, I'd prefer to "involve" rpmbuild in the process so that rpm and yum are "aware" of qmail being recompiled, as I've run into problems otherwise.

 

[madcat]

... bumping

 

[ssiAdmin]

... bumping for madcat...

I'd like to know the answer too.
Any ideas anyone?
Art

 

[atomicturtle]

Sure you can do it with qmail-scanner, it has an archive function. Available in the [atomic] channel from my archive.

 

[madcat]

Thanks for the reply, atomicturtle . I am quite a big fan of http://www.atomicrocketturtle.com/ and am using the yum repositories to keep our servers more up to date. I'm also looking into Atomic Secured Linux.

The problem is that the servers I'm working on have been around for a while, and have email accounts with very weak passwords. I know about the "check passwords in the dictionary" option in the Mail settings in Plesk, but the users have already chosen their bad passwords, so the damage is already done.

The main problem I am encountering is that it appears smtp_auth logins are not being logged unless the login fails. Spammers have no doubt brute-forced a list of working mailnames & passwords a long time ago. Since their logins are not being logged, I cannot tell which accounts are compromised.

Do you know how to either enable logging for all smtp_auth logins (which appears to be the way Plesk was configured prior to version 8), or how to determine which account on the server is responsible for a given outgoing mail message?

Thanks.

 

[atomicturtle]

It would log them by default to either /var/log/messages or /usr/local/psa/var/log/maillog. If its not going to either, then something is wrong with the system. All attempts, successful or otherwise are logged normally.

I know exactly where you are coming from in regard to the weak smtp_auth accounts, we're looking into adding a scanner on the box to look for that in an upcoming ASL update.

 

[madcat]

Solution?

Here's a script I wrote to perform this task. I recommend testing this out on a development server first before installing it on a live production server.

#!/usr/bin/perl

###############################################################################
# qmail-remote Wrapper script
# USE THIS SCRIPT AT YOUR OWN RISK
# This script is provided free of charge, and may be redistributed for free, so long as it
# contains these comments. It is provided on an "as is" basis without warranty of any
# kind, express or implied.
# Suggestions, comments, or bugs can be posted here:
# http://forum.swsoft.com/showthread.php?t=52018
###############################################################################

use strict;
use IPC::Open2;

# Specify path to log file and original qmail-remote binary
my $log_file = "/var/log/qmail-remote.log";
my $qmail_remote = "/var/qmail/bin/qmail-remote.orig";

# Initialize variables
my $lines = 0;
my $max_lines = 25;

# Open a pipe to qmail-remote
my $pid = open2( *QMAILOUT, *QMAILIN, $qmail_remote, @ARGV ) or die "Cannot pipe to '$qmail_remote' : $!\n";

# Open Log file
open( LOG, ">>$log_file" ) or die "Cannot append to '$log_file' : $!\n";
print( LOG "[" . localtime() . "] " . join( ' ', @ARGV ) . "\n" );

# Read all STDIN
while( <STDIN> ) {

# Print to the STDIN of the qmail-remote binary
print( QMAILIN $_ );

# Only print at most the first $max_lines lines of the message to the log file
next if( $lines > $max_lines );

# Print the message to the log file
print( LOG $_ );
$lines++;
}

# Close the STDIN connection to qmail-remote binary so it knows the message is done
close( QMAILIN );

# Get the STDOUT of the qmail-remote binary and print / log it
while( <QMAILOUT> ) {
print( $_ );
chomp( $_ );
print( LOG "* $_\n" );
}

# Print a bunch of dashes to the log
print( LOG "-"x79 . "\n" );

# Close everything
close( QMAILOUT );
close( LOG );
waitpid( $pid, 0 );

Installation instructions (on CentOS 4.6, Plesk 8.4) :

touch /var/log/qmail-remote.log
chmod 662 /var/log/qmail-remote.log

cd /var/qmail/bin
service qmail stop
service xinetd stop
mv qmail-remote qmail-remote.orig
cp <location-of-this-script>/qmail-remote .
chown root:qmail qmail-remote
chmod 555 qmail-remote
service xinetd start
service qmail start

If all goes well, you should see all outgoing messages, including bounces, forwarded messages, messages from PHP & Perl scripts, as well as messages generated via webmail or smtp_auth, in the file /var/log/qmail-remote.log . Each message is separated by a bunch of dashes (ie: "---------------------" ). The first line after the dashes contains a time stamp followed by the parameters passed to qmail-remote by whatever called it. Following that, is the message itself, including headers. At the end, there are a few lines beginning with stars (*) that show the output from qmail-remote.

Hope this helps anyone who is struggling with a spam problem. I discovered that our problem was due to forwarded mail messages, so we have to do something about incoming SPAM too.

Thanks to everyone who replied to this thread.

 

[madcat]

Please note that this forum's "quote" tags did not work exactly as I expected. Some lines in the script show up as two separate lines when they should be on the same line. In addition, I had indented some lines that are not showing up as indented, but indentation is not necessarily important.

 

[madcat]

Loving SpamDyke

Hello everyone. We found ourselves on several anti-spam lists and our mail was "rate limited" by various organizations who found that we were sending "too much SPAM". Initially I had guessed that a PHP / Perl application was compromised and being used to send SPAM, or an easily guessable username & password combination was being used to relay SPAM via smtp_auth.

To make a long story short, it turns out that the source of SPAM was simply due to users setting up forwarding from email addresses on their hosted domains to email addresses at major ISP's. In other words, our servers were not compromised in any way, but were just doing exactly what the end users were asking of it: Forwarding their email to a different provider.

If you find yourself in the same boat, SpamDyke might be the solution for you. IMHO it essentially is a firewall for port 25 (SMTP). It sends a reject signal back to the sending MTA instead of accepting the message and generating a bounce (probably to somebody who didn't actually send the message since the sender address is forged).

For more info, see: http://spamdyke.org/ or http://www.blueoryx.info/linux_administration/spamdyke

Best of all, it actually records the FULL username (not just the part before the '@' symbol) whenever somebody uses smtp_auth to relay messages ... so I actually know who is using the server to relay and from where ... isn't that nice?

 

[Tatsujin]

I have a question regarding login via smtp_auth. Is there a way to tell in the logs who the person authed as? I had an issue earlier with a spammer using a compromised mail account to AUTH smtp and then make up a from address. I was only able to determine where they logged in as (IP Address). With your logger script installed, I did the following test:

[root@nozomu log]# telnet localhost 25
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
220 xxxxx.xxxxxxxx.xxxxx ESMTP
EHLO fakedomain.com
250-xxxxx.xxxxxxxx.xxxxx
250-STARTTLS
250-PIPELINING
250 8BITMIME
AUTH LOGIN
334 VXNlcm5hbWU6
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx <-- real username encoded in base64
334 UGFzc3dvcmQ6
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx <--- real password encoded in base64
235 go ahead
MAIL FROM: [email protected]
250 ok
RCPT TO: [email protected]
250 ok
DATA
354 go ahead
FROM: [email protected]
TO: [email protected]
SUBJECT: you are a champion

you really are dude
.
250 ok 1214752497 qp 25545
^]
telnet> quit
Connection closed.

This is what shows up on your log:

-------------------------------------------------------------------------------
[Sun Jun 29 10:14:57 2008] gmail.com [email protected] [email protected]
Received: (qmail 25548 invoked from network); 29 Jun 2008 10:14:57 -0500
Received: from localhost (HELO fakedomain.com) (127.0.0.1)
  by localhost with SMTP; 29 Jun 2008 10:14:32 -0500
FROM: [email protected]
TO: [email protected]
SUBJECT: you are a champion

you really are dude
* r^@xx.xxx.xxx.xxx accepted message.
* Remote host said: 250 2.0.0 OK 1214752493 a78si5486417pye.19
* ^@

As you can see, since the AUTH LOGIN is encoded, I really have no way of figuring out which user account they used to send out the mail with falsified headers. I wasn't sure if you implemented your log script with this kind of detection in mind; I just wanted to doublecheck.

 

[madcat]

Unfortunately, by default, Qmail on Plesk does not record the username when SMTPAUTH is sucessful, only when it fails. However, if you install Spamdyke, it records the whole username if there is any authentication on port 25, which is great.

For more info, see: http://spamdyke.org/ or http://www.blueoryx.info/linux_administration/spamdyke

Best of all, it actually records the FULL username (not just the part before the '@' symbol) whenever somebody uses smtp_auth to relay messages ... so I actually know who is using the server to relay and from where ... isn't that nice?

 

[atomicturtle]

It logs successful smtp_auth logins on mine to /var/log/messages:

Jun 28 11:52:13 www smtp_auth: smtp_auth: SMTP user scott : /var/qmail/mailnames/atomicrocketturtle.com/scott logged in from [email protected]

This is my syslog.conf for /var/log/messages:

*.info;mail.none;authpriv.none;cron.none /var/log/messages

 

[madcat]

I am using Thunderbird to do this testing, using port 25 and TLS encryption. My syslog.conf entry is the same as yours.

Here's what I get when an login failes, in the qmail maillog:

Jul 8 16:34:16 localhost smtp_auth: SMTP connect from (null)@mydslprovider.com [10.20.30.40]
Jul 8 16:34:16 localhost smtp_auth: smtp_auth: FAILED: - password incorrect () from (null)@mydslprovider.com [10.20.30.40]

The log entries above have been altered slightly. The server's hostname has been changed to 'localhost', the hostname of my DSL provider has been chaned to 'mydslprovider.com' and the IP has been changed to '10.20.30.40'.

With this information, I am able to determine what remote IP generated the failed SMTP AUTH login request, but not what username they tried to use.

For a sucessful login, in the qmail maillog, I get:

Jul 8 16:34:29 localhost spamdyke[12222]: ALLOWED from: [email protected] to: [email protected] origin_ip: 10.20.30.40 origin_rdns: mydslprovider.com auth: [email protected]

In the above log sample, the server's hostname has been changed to 'localhost', the hostname of my DSL provider has been chaned to 'mydslprovider.com' and the IP has been changed to '10.20.30.40'. Also, the username I used to authenticate via SMTP AUTH has been changed to '[email protected]', and the recipient's email address has been changed to '[email protected]'.

I have checked both /var/log/messages and /var/log/secure, and they do not show any entries for either of these SMTP AUTH connection, except for showing the IP address for xinetd.

What type of system are you using, Atomicturtle? I am using CentOS 4 or 5, and Plesk 8.X, and I am finding the same results on all possible combinations of these systems. Note that without Spamdyke installed, a sucessful SMTP AUTH connection is not logged at all.

Thanks.

 

[JLChafardet]

I tried using your perl script mad-cat but it doesnt work, for an odd reason, it started blocking mails instead of sending them, im on the need to log ALL SMTP ACTIVITY our ISP here in my country is behaving oddly and our clients think its a problem related to the server, so I want to start monitoring all smtp activity so if a client tells me "i sent an email at XX hour YY day and it never arrived destination" i can check if the mail actualy was sent using our smtp server or the ISP Server (if the client account never loged on the server for smtp send, then he wasnt using our server to relay the mail).

This is our first major problem with Plesk, and im really worried, as i cannot seem to find any traces of details of smtp messages like "from, to, receive date, deliver date"

any clues on how to acchieve this?

regards,