HOW-TO: finding "Nobody" spammer

[JohnB@]

HOW-TO: finding "Nobody" spammer
==============================

Below is a small script that can be installed in one minute (!) and that I found on the WebHostingTalk forrums:
http://www.webhostingtalk.com/showthread.php?threadid=258294

I corrected several small typos and am re-posting it here. It works on both cPanel and Plesk servers.
At the end of this posting I have some questions one of you might be able to answer.

The script will create a log file to see all activity coming from 'sendmail'. sendmail is being used by PHP spammer scripts. For example, I had spammers on our servers uploading their spammer scripts + database into the /tmp and /dev/shm directories, and this script showed me this ... or showed me the domain of the spammer if he had it in his own domain. The script exchanges sendmail with the script below, so the spammer (and any other form on your server using sendmail) is calling this script instead, which is then logging the user info into a log file before calling the now renamed sendmail. We can now consult that log file for suspicious activity -- namely tons of mails send by "Nobody".
After installation check /var/log/formmail.log to find spammer activity.

Installation:

mv /usr/sbin/sendmail /usr/sbin/sendmail.act
(==>NOTE: Watch out .. if you, by mistake, repeat this install you will overwrite your real sendmail file ... better you make yet another copy with "cp" under another name.)
pico /usr/sbin/sendmail (paste the below code into it)
chmod +x /usr/sbin/sendmail
echo > /var/log/formmail.log
chmod 777 /var/log/formmail.log

------------------------------------------------------------------------

#!/usr/bin/perl

# use strict;
use Env;
my $date = `date`;
chomp $date;
open (INFO, ">>/var/log/formmail.log") || die "Failed to open file ::$!";
my $uid = $>;
my @info = getpwuid($uid);
if($REMOTE_ADDR) {
print INFO "$date - $REMOTE_ADDR ran $SCRIPT_NAME at $SERVER_NAME \n";
}
else {

print INFO "$date - $PWD - @info\n";

}
my $mailprog = '/usr/sbin/sendmail.act';
foreach (@ARGV) {
$arg="$arg" . " $_";
}

open (MAIL,"|$mailprog $arg") || die "cannot open $mailprog: $!\n";
while (<STDIN> ) {
print MAIL;
}
close (INFO);
close (MAIL);

------------------------------------------------------------------------


Now my questions:

(1) If some spammer has uploaded spam scripts into
/tmp, /var/tmp, or /dev/shm
how do I find this guy ... say I see the names of these scripts (before I delete them) ... what other logs can I consult to hopefully find the guy who uploaded them, and what to look for?

(2) What happens if the formmail logfile at /var/log/formmail.log grows real big? I see that other log files at /var/log/ get auto-deleted (after there are 4 of them) and then a new one is auto-created. Is Linux doing this with all files in this folder, or do I need to run some sort of script to do this? (delete and recreate empty file)

Thanks!
John

 

[rtwick]

Originally posted by JohnB
Now my questions:

(1) If some spammer has uploaded spam scripts into
/tmp, /var/tmp, or /dev/shm
how do I find this guy ... say I see the names of these scripts (before I delete them) ... what other logs can I consult to hopefully find the guy who uploaded them, and what to look for?

apache access_log in each vhosts (/home/httpd/vhosts/statistics/logs) should tell you who uploaded it if it is uploaded using a web based script.

If it is uploaded using ftp, then there should also be a xfer log file or something like that at the same location or in /var/log which will tell you who uploaded using ftp.

(2) What happens if the formmail logfile at /var/log/formmail.log grows real big? I see that other log files at /var/log/ get auto-deleted (after there are 4 of them) and then a new one is auto-created. Is Linux doing this with all files in this folder, or do I need to run some sort of script to do this? (delete and recreate empty file)

I think you have to add an entry in /etc/logrotate.conf or something like that or you have to create a file in /etc/logrotate.d/

HTH
R'twick

 

[JLChafardet]

securing /tmp should aid you to avoid things from exec.

do this:

pico -w /etc/fstab
go to :
none /dev/shm tmpfs defaults 0 0

change it to

# none /dev/shm tmpfs defaults 0 0

add a new line down

none /dev/shm tmpfs noexec,nosuid 0 0

Save and exit.

and follow the tutorial

http://eth0.us/?q=node/11
from the Create a 190Mb partition
i know it is for cpanel, but i followed it from

Create a 190Mb partition
start at that step

I would like to thank eth00 on #ev1servers fot helping me on this task!

 

[JohnB@]

A big thanks to both!! This is very helpful, and I am sure this entire thread will also be helpful for many others here.

John

 

[JLChafardet]

the url changed, i edited the post, any way here is it again.

http://eth0.us/?q=node/11

and again thanks to eth00 for making such a great article.


best regards,

 

[frank-sup]

error

Hello

i have secure my tmp file, but when i restart i get error :

update /etc/fstab unexpeded file type for /dev/tmpMnt

i set in :
/dev/tmpMnt /tmp ext2 loop,noexec,nosuid,rw 0 0
in fstab file.

What wrong ??

thanks for help

Frank

 

[alexisb]

How to find spam from [email protected]

Hello, some spammer is sending email from my server, it seems he's using a PHP script.

But he is using [email protected] to send the email, not the user "nobody".

I tried the script in here but didn't get anything logged to the file, but I could see spam getting out from my maillog file.

Any idea how to find this spammer sending from [email protected]?

Thank you.

 

[NightMan]

Originally posted by JohnB
HOW-TO: finding "Nobody" spammer
==============================
John

hi John,
I am not able to edit the senmail file, since it is a binary file. Am I looking in to a wrong file?

 

[atomicturtle]

Side note, the badguys generally beat the noexec flag by just using a script. For example, in one of the more advanced attacks Ive seen, the spammer uploaded a perl based MTA, which was capable of opening outbound connections outside of the local MTA. They initially tried the following:

chmod +x /tmp/spam-script.pl
/tmp/spam-script.pl

which noexec stopped, but then they followed up the attack with:

perl /tmp/spam-script.pl

Which worked of course. The good news here is that this is a little easier to detect when reviewing your logs, by looking for "perl", "sh", and "bash".

 

[NightMan]

Originally posted by atomicturtle
Side note, the badguys generally beat the noexec flag by just using a script. For example, in one of the more advanced attacks Ive seen, the spammer uploaded a perl based MTA, which was capable of opening outbound connections outside of the local MTA. They initially tried the following:

chmod +x /tmp/spam-script.pl
/tmp/spam-script.pl

which noexec stopped, but then they followed up the attack with:

perl /tmp/spam-script.pl

Which worked of course. The good news here is that this is a little easier to detect when reviewing your logs, by looking for "perl", "sh", and "bash".

Thankyou for your sidenote A.turtle!

 

[CyberTech]

I follow the instruction, but there's error, can't use sendmail anymore for php script.
Running on BSD, why?

 

[voodoochile]

This is a large problem with Plesk. I recently brought this up at the ModernBill conference with the Plesk Rep (Todd Crumpler I believe?) and he seemed unaware that there were any such issues. After I told him what was going on, his response was to ask why I didn't just build phpsuexec or something simliar and install it myself . My response was that their updates would break it and it would be a constant PITA.

Part of the problem with the above sendmail wrappers is that as far as I can tell, Plesk uses a non-standard Apache build which strips environment variables. If it's a script being executed as a CGI (pretty much only Perl scripts) you might be able to run it down, however 95% of the abused scripts I've seen are PHP based so when you see 'nobody' sending a ton of spam, it's more thanlikeley a PHP script that's being abused.

Now the fun part, most of the time, spammers don't upload their own spam scripts, they just hammer every form they can find and fill up every field with bccs in order to figure out who is/isn't bounds checking. Once they've figured that out, they can abuse any improperly configured PHP script just as a formmail script with no tracking on the system at all.

Other control panels (cPanel) have a PHP Suexec environment which exectures php scripts as the user that owns the web content, much like current Suexec under Plesk does. This would really make running this stuff down easy.

The last time I contacted sw-soft about a script being abused on our server, it took 4 days and 3 of their techs looking at it to get anywhere, and I'm the one that ended up finding it when it was all done and said. How I did this is the most ghetto damned way possible I think

Essentially I just did an Apachectl fullstatus via the commandline and dumped that to a tmp file. This refreshed every 3 seconds. I was getting hit so often that I could just let this run for a few hours and have a decent log of every GET and POST to my server.

Once I had the big file of all web activity I went ahead and just grepped EVERY damned POST out of the file, sorted them and looked for "Odd" posts. Tons of hit's on guestbooks with no referrers, things like that. After that I had to go run down the access log for that domain, and look for the size of the average POST hitting the script. You'll see an inordinate amount of data being shoved into the script and that's your winner. (if the form takes email address and basic contact info, there's no reason there should be 40K of data being posted to it repeatedly).

This is how you find your spammer. It will always work, but it's not efficient at all. This is one of the major reasons we're currently looking to start transitioning to cPanel, mostly because there's no way this can scale over a large number of servers. This is also not being addressed in Plesk 8 from what I understand. Hopefully they'll get to cracking on it and figure it out.

 

[mikk]

Or you can simply force a unique flag into sendmail for each vhost which will then show up in the return path of the mails (easily viewable with qmHandle)

in vhost.conf (for every domain)

<Directory /usr/local/psa/home/vhosts/DOMAIN/httpdocs>
php_admin_value sendmail_path "/usr/sbin/sendmail -t -i -fXXXXX"
</Directory>


Where XXXXX is a unique identifier

(paths may vary)

(this has been posted a number of times on this forum in the past - i know i have posted it a couple of times)

 

[voodoochile]

mikk, as soon as you have 8-900 domains on a box, it's kind of a pain in the *** to go through insert those, then keep an index and go through and match things up after put your sendmail wrapper into place. Not to mention new accounts getting signed up. Plus you can't run your sendmail wrapper constantly on a high traffic box, so this method relies on you recognizing that your server is spamming, then replaceing the sendmail binary then monitoring for awhile and hoping for your spammer to hit again.

It works, but it's duct tape and this is something that SW-Soft REALLY needs to address. For as large and common as this problem is, there's no reason to be messing with "OMG THE PLESK DESKTOP EXPIERENCE" instead of fixing things like this.

dont' take this as me being pissy towards you, you're in the same boat as the rest of us. This is more of a gripe at SW-Soft.

 

[mikk]

This is not a sendmail wrapper.

It is just forcing the fifth paramter in the php mail() command.

And i do agree its a pain on accounts that are already active. But it is possible, and it enables you to find a spamming php script in minutes - as opposed to the days someone above mentions.

 

[bartje3]

Modified script

Hi,

I simply modified above script to my needs and paths, works fine ! (Plesk 7.5.6 on FC3) Now it's easy to see from where they are sending spam.

Greetings,
Bart

------------------------------------------------------------------------

cd /var/qmail/bin
mv /var/qmail/bin/sendmail /var/qmail/bin/sendmail.real
nano /var/qmail/bin/sendmail

<Now, copy/paste the script below to the window>. Save and Exit.
ls sendmail* (it should return two files, sendmail and sendmail.real ) If not, you didnt' save the fiel in the previous step)

chmod +x /var/qmail/bin/sendmail
echo > /var/log/formmail.log
chmod 777 /var/log/formmail.log

<I would also recommend to restart >
============
Now, all the requests are logged to /var/log/formmail.log, so in order for you to see the content, you can use 'cat' or 'nano' or 'pico' or 'vi', or whatever you want. First see if the log file is created:

cd /var/log/
ls formmail* (you should see formmail.log here. If not, wait a bit and try again. If you still don't see, something is not correct)

cat formmail.log ( Here you'll see the content of formmail.log which should give you a clue on the directory the nobody email was executed from).

Once you find the directory of that email, you can look into the user list and scr#w that user

================
BACKOUT PROCEDURE

In case your script end up not to work, use the following procedure to backout

cd /var/qmail/bin
mv /var/qmail/bin/sendmail /usr/sbin/sendmail.bad
mv /var/qmail/bin/sendmail.real

restart

Now everything should have gone back to normal.

-----------------------------------------------------
Begin script --->>>
-----------------------------------------------------
#!/usr/bin/perl

# use strict;
use Env;
my $date = `date`;
chomp $date;
open (INFO, ">>/var/log/formmail.log") || die "Failed to open file ::$!";
my $uid = $>;
my @info = getpwuid($uid);
if($REMOTE_ADDR) {
print INFO "$date - $REMOTE_ADDR ran $SCRIPT_NAME at $SERVER_NAME \n";
}
else {
print INFO "$date - $PWD - @info\n";
}
my $mailprog = '/var/qmail/bin/sendmail.real';
foreach (@ARGV) {
$arg="$arg" . " $_";
}
open (MAIL,"|$mailprog $arg") || die "cannot open $mailprog: $!\n";
while (<STDIN> ) {
print MAIL;
}
close (INFO);
close (MAIL);

 

[CyberTech]

Not work on Freebsd...

Anyone tested on freebsd?
My sendmail can't sent at all after change it.

 

[NightMan]

have you found a way to rotate this formmail.log?

 

[bartje3]

First you'll have to find out where your real sendmail is located, on FC3 there exists only a symbolic link in /usr/sbin, so the real location is /var/qmail/bin. For freebsd you'll have to change this to the output path of:

which sendmail

To the second reply, no I don't have worked out a rotate yet. I'm more thinking on a better way to filter out all the useless entries. So for example a script could throw out (daily) all entries from the same domain which are below a given limit.
After erasing the useless, the script could email admin a warning message with attached the log.

Greetings,
Bart

 

[serve-you.net]

JohnB:
Thanks for taking the time to post this. It's mostly information from other sources, but it's nice to have it combined into a single source where the people who need it can find it. I personally found it to be useful this weekend in finding a worthless spamming customer.

NightMan:
Just create a file in /etc/logrotate.d with something like:
/var/log/formmail.log {
missingok
notifempty
size 10240k
}

voodoo:
I mean know disrespect, but you have no idea what you are talking about if you think that this is a plesk problem. cpanel users have the same problems with this type of spam, as do Ensim, or any other flavor of the month control panel software. The only way to stop this sort of problem, is to not allow use of the mail command. No joke!

There are 2 different issues here. the first being the casual spammers that search for exploitable mail forms for BCC attacks and the like. The second are your own customers who are spammers, and upload a script to allow them to send mass spam.

Plesk has nothing to do with Apache/PHP. They bundle their own versions for the control panel itself, but the versions your user's web sites are hosted on is whatever version you choose to install. You can do with it whatever you choose, and it is up to you to secure it. Sure Plesk does modify qmail slightly, but in the end, it has nothing to do with whether or not someone can send spam through a web form.

Sadly, our best line of defense right now is to use tools like mod_security to try to block some of the exploitable actions from being able to happen.