HELP! How do I stop QMAIL hijacking by Spammer/HAcker? Calling for Jamesyeeoc!

[criticman]

Quick question for you Jamesyeeoc:

Someone, supposedly from Brazil, has managed to flood my server with attempts to send spam.

I have gone through what you have said in other posts, as well as ART's posts. I have installed qmHandle. I had 400,000+ yesterday that I deleted with qmHandle.

I have set queuelifetime at 50,000.

I have added the "from" address to badmailfrom, but since they are spoofing it keeps getting through. Although, it still appears the same e-mail address is being used in the "From" .

GoDaddy has shut down my outgoing mail traffic, but the "hacker/spammer" is able to keep attempting the sending of mail.

Here is a header from a spam message:

 --------------
MESSAGE NUMBER 157508509 
 --------------
Received: (qmail 27782 invoked by uid 2520); 17 Dec 2005 11:48:27 -0700
Received: from 127.0.0.1 by lonetreehosting.prod.phx1.secureserver.net (envelope-from <[email protected]>, uid 48) with qmail-scanner-1.25st 
 (clamdscan: 0.87.1/1211. perlscan: 1.25st.  
 Clear:RC:1(127.0.0.1):. 
 Processed in 0.150098 secs); 17 Dec 2005 18:48:27 -0000
Date: 17 Dec 2005 11:48:27 -0700
Message-ID: <[email protected]>
content-type: text/html
Subject: URGENTE: Verifique seu CPF
From: [email][email protected][/email]
To: [email][email protected][/email]
X-Qmail-Scanner-1.25st: added fake MIME-Version header
MIME-Version: 1.0

GoDaddy is sucking presently at fixing this. I am trying to figure it out, but I am not a mail admin pro by any means, so please help.

 

[criticman]

Oh, and I also went through the mail logs yesterday. I could not find a false SMTP connection made at any point yesterday afternoon or evening when this began occurring. All logged SMTP connections were from my valid users and demonstrated that they did valid SMTP auth.

I did check and see what IP's were sending the most traffic through, GoDaddy blocked the biggest ones, but still happening.

I also have set each site, through Plesk, to just "Reject" mail to nonexistent users instead of bouncing it.

UPDATE: Also, I have been using the following command to delete the bulk of the spam attempts since, from what I have seen, they have the same message subject:

 ./qmHandle -SURGENTE

 

[criticman]

Stupid GoDaddy has no clue. They let me know an issue occurred way back in September - how, I do not know as I did not open up any ports or undo any security settings, so their initial config was not proper. Even so, they want me to back up "all essential data" so they can wipe it clean. Like I have time (or my clients) to let it all get wiped then reconfigured.

Any advice would be great. A copy of their message is below.

This email is in response to your recent customer service inquiry regarding Virtual Dedicated Hosting.

After researching the issue(s) at hand, we have determined the following:
Your Virtual Dedicated server has been compromised on or around 9/9/05. Attackers loaded via Apache the file: enviar.pl - a mass mailing linux script. The Apache logs only go back to November, at this time we are unable to determine root cause. We have chmod 000 all identified malicious content. Please back up all essentail data so we may initiate a reprovision (complete wipe) or your Virtual Dedicated server. Please contact us if you have any questions and when you are ready for the reprovision.

Location of SPAM message (corpo.html), the SPAM script (enviar.pl) and email lists:

bash-2.05b# pwd
/tmp/.teste
bash-2.05b# ls -la
total 25102
drwx------ 2 apache apache 1024 Dec 17 03:17 .
drwxrwxrwt 5 root root 10240 Dec 17 13:12 ..
-rw------- 1 apache apache 4574 Dec 12 00:40 corpo.html
-rw------- 1 apache apache 1106 Sep 9 19:33 enviar.pl
-rw------- 1 apache apache 15305007 Dec 12 00:13 lista.txt
-rw------- 1 apache apache 10325082 Dec 17 03:17 lista2.txt



Should you require further assistance on this or any other issue, don't hesitate to contact us any time of the day or night at (480) 505-8877. Or, if you prefer email, you can send your questions or comments to [email protected].

We appreciate your business and endeavor to provide the utmost in customer service and support.

 

[SecondPhase]

Hi there, i'm not an expert here but i'll throw out some idea that i've read recently around this forum..

I would guess that a hacker installed this spammer mailer tool on your system using a PHP or Apache security hole probably using the wget command to get this stuff in your /tmp directory.

First type from the command line:
chmod 700 /usr/bin/wget
so only the root user can run this.

Then search the forum to install / configure Apache mod_security and PHP patches or inits that are more secure..

I guess it depends whether you think anything else was done to the machine. GoDaddy probably doesn't want to risk it and best to just wipe it out in case other features are compromised (backdoor?) or have virus.. somebody else here may know from the exact description of your problem what else may have happened..

If it's just a spamming script, deleting it and closing the security hole may be ok for you.. maybe the wget permission will solve it by itself, but i've been reading about other security measures and will be implementing some more of them myself.

 

[criticman]

Well, their efforts were not helpful.

After their email, I deleted the spam messages from the queue by subject.

Two hours later, 10,000 more in there.

Please help!

 

[criticman]

I just chmod'ed wget, chmod 700 /usr/bin/wget

Will delete queue and see, but I am guessing there is another script on there.

UPDATE: I do not just trust their changing permissons on those files, so I deleted the spam script and the directory that housed it.

Hoping it will work.

Wow, I feel like I am just talking to myself here. If someone wants to chime in, much appreciated.

 

[jamesyeeoc]

Sorry I didn't see this earlier, I've been sick with the Flu. Some of the following may not be in good order, still can't think too well due to sick and medications....

1. GoDaddy will most likely not be of much use, other than a reimage.

2. They are just now informing you of a malicious script from September????

3. WGet is not the only way they can upload files to your server, so chmodding wget is of small use, but not a 'cure all'.

4. Check the /tmp for any other files or hidden dir's (such as .tempe), there may be more than just one.

5. If they do a server reload, the FIRST THING TO DO before reloading any hosted domain files is to SECURE YOUR SERVER! You should check into ARTs ASL (Atomic Secured Linux) but at the very least install mod_security, your AV of choice, Spamassassin, AND qmail-scanner (also from ARTs site). Also install RKHunter and CHKRootkit and have them cron'd to update and then run scans, optionally email results to you.

6. Make sure that any exploitable packages (such as phpBB) are updated on your client's sites. All it takes is for one bad package to be installed and it will compromise the entire server.

7. There is no easy or simple fix.

8. After deleting the script/directories, YOU MUST MAKE SURE THERE ARE NO PROCESSES RUNNING IN MEMORY which are related to the malicious script. Review your process list carefully.

9. Install and configure mod_security. The default settings only include a couple of the available rulesets, you may need to add some of the additional ones.

10. All this 'assumes' that it is most likely an exploit, not a hack/crack or rooting. If I get more coherent and think of anything else I'll post again.

11. Checking logs for SMTP connections is useless since the emails are originating from within your server (the malicious script), not from 'outside' your server.

 

[criticman]

Awesome, you posted! Sucks that you are sick - I hope you get better soon.

I did most of your steps above prior to them telling me they were going to wipe it. I was in the process of installing and configuring mod_security, chkrootkit, and rkhunter - found good info on setting it all up thanks to you all on the forums.

I have backed it all up and told GoDaddy that it is ready for them to wipe. Their claims that it would be quick changed, "process will be taken care of in 24-48 hours."

But yes, once they wipe it, I will setup the security provisions prior to putting client sites back on there.

I think I may invest in setting up a secondary server with Godaddy, so if this happens again I can mod DNS to point to it while the other one is down. Think this is a worthwhile investment? Sure beats stepping up to "the big boy" hosting, such as rackspace...although with them it would never have happened and I could've finished several PHP projects this weekend.

 

[phatPhrog]

mod_security

In RE: to jamesyeeoc

9. Install and configure mod_security. The default settings only include a couple of the available rulesets, you may need to add some of the additional ones.

You should go to http://gotroot.com and grab the up-to-date rules written there. (thank you ART and assoc.)

Also, there is a script available at gotroot that will automatically update the mod_security rules. I edited and added to it so all of the available rules are updated.

The original script checks and rolls back to last updates if the new ones cause apache problems.

You can run the update script manually or set it up as a cron.

If, after you grab the original script from gotroot you'd like to have my edited version, let me know and I'll send it.

Works great for us.

jamesyeeoc - Do hope you are feeling better

 

[criticman]

Well ****....so I was rushing to get it backed up....and I failed to make the shell map file and IP map files.

So, any quick way to fix this? I found the format on PSA website. I will just have to manually configure my entire configuration (luckily only 29 domains) unless someone has another way.

 

[phatPhrog]

Originally posted by criticman
Well ****....so I was rushing to get it backed up....and I failed to make the shell map file and IP map files.

So, any quick way to fix this? I found the format on PSA website. I will just have to manually configure my entire configuration (luckily only 29 domains) unless someone has another way.

The map file is created in the restore process:

Just run:

cat dump.* | /usr/local/psa/bin/psarestore -m map_file -s shells_map_file -f -

then run

/usr/local/psa/bin/psadump -F -f $backupdir/dump2_$date.svr

in whatever name convention you use for your backups.

 

[criticman]

Originally posted by francuz
To be able to create shell mapping file you need to have some dump file. After that procedure is the following:

/usr/local/psa/bin/psarestore -t -f <your_existing_dump_file> -s shells_map_file -m ip_map_file

-t - test mode, if this key is specified then restoring procedure will not run, only the checking process;

This command will create two files in current directory: shells_map_file, ip_map_file

I did this, and bingo, made the files. When I try to run the restore now using those files, it is throwing this error:

Read file with ip map and check system ip addresses:

Target ip '68.178.166.17' is found in the system on interface 'venet0' with mask: '255.255.255.255'
Syntax error in ip map file in line 31: 68.178.166.1 ->

 

[criticman]

Here is the ip_map_file contents:

# ATTENTION: all not existing TARGET ip addresses will be added into your system.
# Write target ip addresses for mapping.
# for example:
#
# fxp0:
# 192.168.1.1 -> 195.173.23.200
# eth0:
# 192.168.1.2 -> 193.233.1.130
# 192.168.1.3 -> 193.233.1.131
#
# or simply:
# 192.168.1.1 -> fxp0:195.173.23.200
# 192.168.1.2 -> eth0:193.233.1.130 255.255.255.0
# 192.168.1.3 -> eth0:193.233.1.131 255.255.255.0
#
# 1) if ip address exists in the system then it will be skipped on the old interface.
# 2) if netmask was not specified then used 255.255.255.0





#########################################################################
# The following ip addresses are used for domains:
#########################################################################

# used for domains: morgantheatrical.com, kemteclabdev.com, liveoakcavespring.com, look2find.info, nononotever.com, palmettobean.com, hotrodconstruction.com, theidentitygroup.us, thaicoon.net, lecroycompany.com, thehamptonscondos.com,$
68.178.166.17 -> venet0: 68.178.166.17 255.255.255.255 ### shared

# used for domains: lonetreehosting.com, criticman.com, look2find.com
68.178.166.1 -> 255.255.255.0 #<-uncomment this netmask to use it as a template ### exclusive

# used for domains: admissionsannex.com, daviesshelter.com, wholesalecoffeenetwork.com, drmattbynum.com, bobbyjcobb.com, palmetto-trust.com, safetysouth.com, mardigrascharityball.com, sjcatholicschool.org, scthespians.org, sjhigh.org,$
68.178.166.16 -> venet0: 68.178.166.16 255.255.255.255 ### shared



#########################################################################
# The following ip addresses are not required:
#########################################################################

 

[phatPhrog]

First, how did this topic end up being about backup/restores? Thought you were having problems with hijacking?

Hope we don't get yelled at for straying off subject here.

Anyway, I'll close up here by saying we run a script for our backup/restores, and the following map creation and restore works without error for us.

If you run the following first;

cat dump.* | /usr/local/psa/bin/psarestore -m map_file -s shells_map_file -f -

then;

/usr/local/psa/bin/psarestore

FROM the directory where your dump file resides you should have no problem with a restore.

This of course depends on whether you have a working server. The dump doesn't backup the entire server.

We're gonna get smashed by continuing this conversation off topic.

Believe me, I know; 'cause my history proves off-topic subjects.

Just trying to help.

 

[criticman]

Heh. I started the topic, so people can get bent if they feel the need to complain. This is all an ongoing issue I am trying to get help with.

Server is indeed working. So that method will create the needed map files that actually work? I will try, because I keep getting the same error with the info used above.

 

[criticman]

No luck:

# cat dump.* | /usr/local/psa/bin/psarestore -m map_file -s shells_map_file -f -
cat: dump.*: No such file or directory

===============================================================================

Attempt to connect to MySQL server ... done

===============================================================================

Checking packages installation ... done

===============================================================================

Dump reader error: Input file is not a valid MIME file: can not find MIME header.

 

[criticman]

Tweaking more, now getting this issue....

Read file with ip map and check system ip addresses:

Target ip '68.178.166.17' is found in the system on interface 'venet0' with mask: '255.255.255.255'
Target ip '68.178.166.1' is found in the system on interface 'venet0' with mask: '255.255.255.255'
Error encountered in the map file: exclusive source ip address '68.178.166.1' with owner 'mwilliams' can't be mapped to exclusive ip address '68.178.166.1' with owner 'mrwilliams'
Target ip '68.178.166.16' is found in the system on interface 'venet0' with mask: '255.255.255.255'
Errors in ip mapping procedure.

UPDATE: GOT IT! Typo...I had username with initial on one...removed, all good now...or at least it is running the restore!

 

[criticman]

So apparently the key they put into plesk when they wiped it was not proper. So, copied restored key over, then going back and redoing the restore after I reboot....

 

[criticman]

DONE!

Thanks everyone for your help! Time to secure the **** out of the server even more!

 

[nhouse]

Hello everyone... I am unfortunately having the same issue that was stated in the beginning of this thread. Only my spammer comes across as [email protected] and another one or two varients with similar addresses. criticman, did you or anyone else here find out an answer from GoDaddy or elsewhere? I did some Googling and came up with a rederence or two to it possibly being the result of a virus. See these links to review:
Symantec Site
Trend Micro Site

Ok, with that said... if one of my clients has a virus on their system (maybe one at home, etc.) then I suppose it couple be possible that the virus has somehow "grabbed" their smtp server settings and is mass mailing from their machine. Does this sound logical? I do have a virus scanner on my server (virtual dedicated) but I also guess that it wouldn't catch outbound virus / spam activity.

Anybody found similar types of issues or maybe a solution? Also, can you give me tips on how or what to look for in the log files that would assist me. So far I can't spot anything that looks off. I see lots of these:

Feb 26 12:14:07 nhousemedia qmail: 1140977647.369362 starting delivery 6393: msg 227648526 to remote [email protected]
Feb 26 12:14:07 nhousemedia qmail: 1140977647.369879 status: local 0/120 remote 117/120
Feb 26 12:14:07 nhousemedia qmail: 1140977647.370403 starting delivery 6394: msg 227647928 to remote [email protected]
Feb 26 12:14:07 nhousemedia qmail: 1140977647.370883 status: local 0/120 remote 118/120
Feb 26 12:14:07 nhousemedia qmail: 1140977647.371472 delivery 6383: success: 64.202.189.86_accepted_message./Remote_host_said:_250_ok_1140977646_qp_23890/
Feb 26 12:14:07 nhousemedia qmail: 1140977647.371959 status: local 0/120 remote 117/120
Feb 26 12:14:07 nhousemedia qmail: 1140977647.372446 end msg 227648512

Can you folks spot any obvious things here? It looks like they are being authenticated by someone... but who? Is there a way to find out the IP address of the sender... or if it local, the exact email user???

Any help would be VERY much appreciated. This **** has happened twice over the weekend with one occasion the queue ranging up to 155,000 messages in the remote and 22,000 in the local... this killed xinietd and qmail crashed.