SPAM PROBLEM, please help me!!!!!!!

[disvirtual]

Hi.

I have a serious problem with my web/mail server.

The problem is that Qmail queue fills of Spam at certain day periods.

The spam attack begins and queue grows VERY fast. You can imagine the problems that it’s causing: mails delayed for hours, mailboxes full of spam, sever processing load, and of course, clients phoning and complaining about that.

I noticed that when I stop the web server (httpdd) the spam incoming ceases instantaneously, so conclusion is immediate: the attack comes from my own server, a web, a PHP script or any other kind of scrip susceptible of having any kind of exploit.

Almost all spam messages are ONE SENDER TO ONE RECIPIENT, so qmail-qread doesn’t help to locate sender-victim.

Spamassassin does NOTHING but mark messages as spam (TOTALLY USELESS THING), so server loads and mailboxes fulls of –marked- spam.

I’m sure open relay is off (http://kb.odin.com/en/1394)

I created wrapper for sendmail to log all mail sending from web scripts (http://kb.swsoft.com/en/1711). Result = NOTHING.

I followed article http://kb.odin.com/en/766 and nothing.

The command line in the same article:
# lsof +r 1 -p `ps axww | grep httpd | grep -v grep | awk ' { if(!str) { str=$1 } else { str=str","$1}}END{print str}'` | grep vhosts | grep php

Shows NOTHING, even if I launch a PHP script for intensive mail test.

The only thing I achieved with that procedure is the following:

Viewing queue messages in Plesk, I took the sender and “cat /usr/local/psa/var/log/maillog |grep USERNAMEâ€.
Obtained: qmail: 1211215564.841050 info msg XXXXXXX: bytes 1713 from <[email protected]> qp 19251 uid 2020

Then find /var/qmail/queue/mess/ -name XXXXXXX
Then cat /var/qmail/queue/mess/YY/XXXXXXX

Some show: (qmail 9355 invoked from network)
others show: (qmail 9355 invoked from 110)

if “grep 110 /etc/passwd†It shows: “popuser:x:110:31OP3 service user:/var/qmail/popuser:/sbin/nologinâ€

In an external link (http://www.atomicorp.com/wiki/index.php/Spam) I found: “If the userid is popuser, the source is a compromised smtp_auth accountâ€,

And now the question: HOW CAN I FIND THE USER/SCRIPT/WEB/IP OR WHATEVER IS SPAMMING MY SERVER ????????????

I have read all those forum posts:

http://forum.swsoft.com/showthread.php?t=16774
http://forum.swsoft.com/showthread.php?t=47861
http://forum.swsoft.com/showthread.php?t=48999
http://forum.swsoft.com/showthread.php?t=44012
http://forum.swsoft.com/showthread.php?t=52018
http://forum.swsoft.com/showthread.php?t=52759
http://forum.swsoft.com/showthread.php?t=52550
http://forum.swsoft.com/showthread.php?t=3791
http://forum.swsoft.com/showthread.php?t=52113
http://forum.swsoft.com/showthread.php?t=51738
http://forum.swsoft.com/showthread.php?t=51577
http://forum.swsoft.com/showthread.php?t=51466
http://forum.swsoft.com/showthread.php?t=48264
http://forum.swsoft.com/showthread.php?t=49750

And many others, and lots of KB articles, but none helped me.

Please, I NEED VERY URGENT HELP !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Thanks a lot.

 

[disvirtual]

Pop/imap Spam

Now the spam attacks are comming from POP/IMAP server.

When I stop POP service spam stops incoming, so seems obvious that spam comes through POP server.

How can it be? a hacked user password? some kind of exploit? what must i do? how can I found the compromised account (if any)?

Help me, please, I'm really desperate.

 

[mediashaker]

When you go:

1. /var/qmail/bin/qmail-qread
18 Jul 2005 15:03:07 GMT #2996948 9073 <[email protected]> bouncing
done remote [email protected]
done remote [email protected]
done remote [email protected]

2. find /var/qmail/queue/mess/ -name 2996948

3. cat /var/qmail/queue/mess/22/2996948

4. Look Up the ip address of the connected user in the email message headers

5. grep ipaddressofhackedaccount /usr/local/psa/var/log/maillog

Also you can view authenticated users just by typing:
grep smtp_auth /usr/local/psa/var/log/maillog

Also note that the maillog gets archived fairly often so you might need to gunzip and search through the archived logs ..

 

[disvirtual]

Thanks, but no way.

doing: #tail -f /usr/local/psa/var/log/maillog | grep smtp_auth

I got a realtime maillog entries containing smt_auth. They look all like that:

smtp_auth: smtp_auth: SMTP user : logged in from (null)@n234.cpms.ru [87.236.29.234]

so no known user shown.

 

[disvirtual]

ah, obviously that is a fake IP.

 

[Spazholio@]

http://private.dnsstuff.com/tools/ipall.ch?domain=87.236.29.234

Looks valid to me, man. And if I might make a suggestion, when asking for help, and someone offers you help - don't say "Thanks, but no way." It comes across as unappreciative, even if it's not meant that way.

 

[mouse]

Null User Spam attack

I have been plagued by several spammers / spam bots that have been injecting spam into qmail via a null user login

# cat /usr/local/psa/var/log/maillog |grep null
Jun 17 09:02:46 penguin4 smtp_auth: SMTP connect from (null)@13.224.136.219.broad.gz.gd.dynamic.163data.com.cn [219.136.224.13]
Jun 17 09:02:46 penguin4 smtp_auth: smtp_auth: SMTP user : logged in from (null)@13.224.136.219.broad.gz.gd.dynamic.163data.com.cn [219.136.224.13]

I have spent alot of time researching this over the last month and belive I may have an answer for those that want to stop this before the patch of 8.4.1.

let me continue with a bit more information before I give conclusion

Plesk Control Panel version
psa v8.4.0_build84080514.18 os_FedoraCore 6
Operating system
GenuineIntel, Intel(R) Xeon(R)CPU 5130 @ 2.00GHz
Linux 2.6.18-1.2798.fc6

My first move was to add the domain 163data.com.cn to the blacklist.
I have never seen anything but spam come from 163data anyway.
So thought all was solved for the server until the next day when sure enough 100's more spam from 163data.com.cn.
So next step was to add

ALL: .163data.com.cn : DENY
to /etc/hosts.allow
this was sure to get these buggers

Well come the next few days things seemed ok
then bam! they where back again and they sure had me baffled
so this had to be a user or internal was my thoughts
at which point I looked at the smtp-auth, discovered the null logins
and also noticed that they where using my reverse.DNS names as the senders name,
I thought this curious and noted that this was a way to get thru the hosts.allow block
I proceeded to recreate how they where getting thru and sure enough a simple

telnet MYDOMAIN.com 25
smtp_auth: AUTH [email protected]
smtp_auth: PASS (null)

got me thru
WOW how could this be?

I tried removing the reverse DNS in the thoughts that relaylock would pick it up
(NOPE didn't work)
Was so frustrated that I decided to move all the server IPs in the hope of at least tracking why (really didn't want to do this)
It was when I got to the point of actually moving the IP's did I notice one common denominator.
The IP's that they where using for the names (reverseDNS names) had NO SITES ON THEM
I had added a block of IP's in preparation for several sever migration moves and never needed all of them but just left them in-place for future additions.

at this point I have removed unused ones and added a site to one they picked on the most
- after 3 days I see their failed null attempts but the spam is gone
hope this helps

Jerry The Mouse

 

[brucekurz]

I have had this issue as well and I made a file called badmailfrom in /var/qmail/control and added :
(null)@(null)
null@null
(null)@
(null)@*
null@
NULL@
(NULL)@(NULL)
(NULL)@
(NULL)@*
*@(null)
*@(NULL)

Above is every variation and wildcard for the people that have been spamming. So far for the past few days the server for mail has been thankfully quiet. I found the idea of the wild card and it stopped it right away. I still hope for thepatch to come very soon.
http://www.psoft.net/rus/HSdocumentation/sysadmin/qmail_configuration.html#badmailfrom

I hope this may help some of those who have spent endless hours like myself looking for bad scripts to disable only to find hundres of emails of spam in the queue.

 

[atomicturtle]

That "null" means that they're logging in using base64 encoding on the username field. The SMTP_AUTH code isnt set up to decode that, so it uses "null". What you need to do to figure that out is to fire up a sniffer on port 25 and catch what they're sending, then decode that. I'm sure you'll find its something very simple like "info", "www", or "webmaster" with an equally simple password.

 

[brucekurz]

well that did not work what i posted last night. We were just hit again. any suggestions on a sniffer to find out which login they are using to stop this.

 

[Ragefast]

ngrep is your friend, in this matter.

ngrep -q host <ipofspammer> port 25

It will render you something like:

T <spammer>:18701 -> <server>:25 [AP]
AUTH LOGIN..
#
T <server>:25 -> <spammer>:18701 [AP]
334 VXNlcm5hbWU6..
#
T <spammer>:18701 -> <server>:25 [AP]
dXNlcm5hbWU=.. <-- this is the base64 encoded username
#
T <server>:25 -> <spammer>:18701 [AP]
334 UGFzc3dvcmQ6..
#
T <spammer>:18701 -> <server>:25 [AP]
cGFzc3dvcmQ=.. <-- this is the base64 encoded password

You can then decode that base64 login, here's a good place to do that:

http://makcoder.sourceforge.net/demo/base64.php

 

[aaargh]

Hi all,

I've got a similar problem, 1 server is flooded with 120000+ spam mails in the queue
After detecting, I used qmHandle to clear the mailq.
I've eliminated all known sources ( open-relay, faulty cgi/php scripts, etc.. )
Checked for "smtp_auth" entries in the mail.log ( there are none )
After checking some spam headers they all have the following lines in the header:

Received: (qmail 16677 invoked by uid 110); 17 Oct 2008 23:35:23 +0200
Received: (qmail 16634 invoked from network); 17 Oct 2008 23:35:23 +0200

uid 110 = popuser

even tried the 'ngrep' command, like this: "ngrep port 25 and src host 205.209.161.84"
which shows me there is no smtp authentication used !

filter: (ip or ip6) and ( port 25 and src host 205.209.161.84 )
#########
T 205.209.161.84:4251 -> 195.110.22.25:25 [A]
......
#
T 205.209.161.84:4252 -> 195.110.22.28:25 [A]
......
#
T 205.209.161.84:4253 -> 195.110.22.28:25 [A]
......
#
T 205.209.161.84:4254 -> 195.110.22.29:25 [A]
......
#
T 205.209.161.84:1742 -> 195.110.22.29:25 [A]
......
#
T 205.209.161.84:3350 -> 195.110.22.30:25 [A]
......
#
T 205.209.161.84:4802 -> 195.110.22.33:25 [A]
......
#
T 205.209.161.84:4111 -> 195.110.22.43:25 [A]
......
#
T 205.209.161.84:4253 -> 195.110.22.28:25 [AP]
EHLO msg-g09pmirpcam..
#
T 205.209.161.84:4254 -> 195.110.22.29:25 [AP]
EHLO msg-g09pmirpcam..
#
T 205.209.161.84:4251 -> 195.110.22.25:25 [AP]
EHLO msg-g09pmirpcam..
#
T 205.209.161.84:4252 -> 195.110.22.28:25 [AP]
EHLO msg-g09pmirpcam..
#
T 205.209.161.84:4802 -> 195.110.22.33:25 [AP]
EHLO msg-g09pmirpcam..
#
T 205.209.161.84:3350 -> 195.110.22.30:25 [AP]
EHLO msg-g09pmirpcam..
#
T 205.209.161.84:1742 -> 195.110.22.29:25 [AP]
EHLO msg-g09pmirpcam..
#
T 205.209.161.84:4111 -> 195.110.22.43:25 [AP]
EHLO msg-g09pmirpcam..
#
T 205.209.161.84:4253 -> 195.110.22.28:25 [AP]
RSET..MAIL FROM:<[email protected]>..RCPT TO:<[email protected]>..

The lucky side of this spam run is that it is addressed to only one remote domain, so i've added it to Plesk with a catch-all account, so right now i have over 50K of evidence mails in the mailbox.
The sender hosts are now rejected by firewall, but new hosts keeps popping up ( and still only send to that particular remote domain.)

Please help me to find a real solution to stop this.

Maurice

 

[brucekurz]

I hope this helps

I actually found 2 things today that may have solved it. 1 was do this.
updatedb
then I saw that a script dm.cgi was running and had 100's of items in top and server cpu was going very high. I located the dm.cgi and found that they had hidden txt log files and it showed thousands and thousands of email being sent, bad email addresses and the minute I remove that cgi access for that user I also changed ftp info on the domain and removed domain admistrator access. I contacted the user and they said they hired someone from the uk gave them ftp and Domain Administrator access and never changed this info so they were ftp'ing and spamming like crazy every day.

I hope this helps.
Bruce

 

[brucekurz]

ps. google of dm.cgi can cause major problems cpu, sql stop and server crashes. I also opened the dm.cgi and it was all encrypted.
Delete it if you find it.

 

[aaargh]

thanks for the help
But no weird cgi scripts were found on the server.
and if it was an cgi script, the "revoked uid" shouldn't be 110 ( popuser )

I'm still in the dark...

 

[kuhle]

Similar problem - can you help me find the source?

I have recently migrated to a new server running Plesk 8.6, and there have been some Spam emails going through the mail server. Can anybody help me identify where it originates please?

[root@plesk2 ~]# /var/qmail/bin/qmail-qstat
messages in queue: 5
messages in queue but not yet preprocessed: 0
[root@plesk2 ~]#

[root@plesk2 ~]# cat /usr/local/psa/var/log/maillog |grep -I smtp_auth |grep -I user |awk '{print $11}' |sort |uniq -c |sort -n
152 logged
[root@plesk2 ~]#

[root@plesk2 ~]# /var/qmail/bin/qmail-qread
23 Mar 2009 12:38:02 GMT #14814946 674 <[email protected]> bouncing
done remote [email protected]
done remote [email protected]
done remote [email protected]
done remote [email protected]
done remote [email protected]
done remote [email protected]
done remote [email protected]
done remote [email protected]
done remote [email protected]
done remote [email protected]
done remote [email protected]
done remote [email protected]
done remote [email protected]
done remote [email protected]
done remote [email protected]
done remote [email protected]
done remote [email protected]
done remote [email protected]
done remote [email protected]
done remote [email protected]
done remote [email protected]
done remote [email protected]
done remote [email protected]
done remote [email protected]
done remote [email protected]
done remote [email protected]
done remote [email protected]
done remote [email protected]
done remote [email protected]
done remote [email protected]
done remote [email protected]
done remote [email protected]
done remote [email protected]
done remote [email protected]
done remote [email protected]
done remote [email protected]
done remote [email protected]
done remote [email protected]
done remote [email protected]
done remote [email protected]
done remote [email protected]
done remote [email protected]
remote [email protected]
done remote [email protected]
done remote [email protected]
done remote [email protected]
done remote [email protected]
done remote [email protected]
done remote [email protected]
done remote [email protected]

[root@plesk2 ~]# find /var/qmail/queue/mess/ -name 14814946
/var/qmail/queue/mess/2/14814946
[root@plesk2 ~]#

[root@plesk2 ~]# cat /var/qmail/queue/mess/2/14814946
Received: (qmail 19790 invoked from network); 23 Mar 2009 12:38:02 +0000
Received: from marugoto-5-210-157-013-203.interq.or.jp (HELO User) (210.157.13.203)
by xxx.xxx.xxx.xxx (my IP) with SMTP; 23 Mar 2009 12:38:02 +0000
From: "BBVA.net"<[email protected]>
Subject: BBVA.net lanza su nueva Promocion | "100 euros te esperan"
Date: Mon, 23 Mar 2009 07:37:47 -0500
MIME-Version: 1.0
Content-Type: text/html;
charset="Windows-1251"
Content-Transfer-Encoding: 7bit
X-Priority: 3
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook Express 6.00.2600.0000
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000

<a href="http://besstek.com/.es/"><img src="besstek.com/.es/2.gif"></a>

[root@plesk2 ~]# grep 210.157.13.203 /usr/local/psa/var/log/maillog
Mar 23 12:20:07 plesk2 relaylock: /var/qmail/bin/relaylock: mail from 210.157.13.203:34978 (marugoto-5-210-157-013-203.interq.or.jp)
Mar 23 12:20:12 plesk2 smtp_auth: SMTP connect from (null)@marugoto-5-210-157-013-203.interq.or.jp [210.157.13.203]
Mar 23 12:20:12 plesk2 smtp_auth: smtp_auth: SMTP user info : logged in from (null)@marugoto-5-210-157-013-203.interq.or.jp [210.157.13.203]
Mar 23 12:21:11 plesk2 relaylock: /var/qmail/bin/relaylock: mail from 210.157.13.203:35024 (marugoto-5-210-157-013-203.interq.or.jp)
Mar 23 12:21:14 plesk2 smtp_auth: SMTP connect from (null)@marugoto-5-210-157-013-203.interq.or.jp [210.157.13.203]
etc.
etc.
etc.
[root@plesk2 ~]#

and I do not appear to have ngrep:

[root@plesk2 ~]# ngrep port 25 and src host 210.157.13.203
-bash: ngrep: command not found
[root@plesk2 ~]# ngrep
-bash: ngrep: command not found
[root@plesk2 ~]# ngrep -q host 210.157.13.203 port 25
-bash: ngrep: command not found
[root@plesk2 ~]#

Is there a way i can try to find out where this Spam is originating, and how I can stop it? I really hope that someone can advise with the above info. Thanks in advance.

 

[atomicturtle]

I'll bet your "info" user has a really simple password. Like 12345 or something

 

[DaveK]

Yes!! ngrep to the rescue

Thanks to all on this thread. Using ngrep I was able to catch him in the act.

The base64 link above wasn't working. This one did:
http://www.opinionatedgeek.com/dotnet/tools/Base64Decode/

Also this blog post: 'Easily Find Bad Email Passwords on Plesk' actually found the bad account / password used by the spammer and made it easy to quickly clean up a bunch of other potential problem accounts:

http://www.rackaid.com/resources/easily-find-bad-email-passwords-on-plesk-1/