Server compromised?

[lpittman]

Hey team ...

I installed isoqlog to see what the usage was like on my server and was alarmed to see 6500+ "Sent" emails after scanner only todays log.

I run 'isologd' every 10 minutes or so just to see and the Sent emails increases by a nearly 100 every time. This mail server _does not_ send out that many legitimate outgoing emails.

So, I have two thoughts:

1. I do have qgreylist running - so, technically if I am receiving a lot of spam, this would be sending back a lot of emails, right?

2. The server has been compromised.

I've checked /tmp and /var/tmp and see nothing out of the ordinary. I have run chkrootkit and rkhunter and neither reports anything strange.

Can anyone offer any advice here?

Thanks,
Luke

 

[faris]

qgreylist does not send out any emails to incoming spam. The "come back later" type "message" is actually part of the handshake if you like. So it won't generate any messages at all when it is graylisting.

So yes, someone could have found a way to compromise a script on the server, or could have installed a spamming script. But it could also be one of your customers who is accidentally doing this.

For example one of our customers had a mailserver at their side that was compromised. Unfortunately it was set to use the smtp service on their account on our servers. So the spam went out via us rather than them (though we found the problem and dealt with it before too much harm had been done).

It could also be one of your customers is being spammed to high heaven, and has their account set to reply to emails sent to non-existant users or something like that.

As a side note I have noticed that spammers have been particularly active over this weekend. This is unusual - normally the peak is on Thursdays in my experience.

Anyway, I don't know what else to suggest you really...if you have mod_security installed take a look through the log to see if any particular site has been probed a lot, or if any particular IP has been probing, then go through the website logs to find a match maybe).

Also do the usual rkhunter type stuff to make sure nothing serious has happened.

Faris.

 

[lpittman]

Howdy,

Thanks for the reply.

I should have realized qgreylist doesn't send emails back - good point.

I've run rkhunter and chkrootkit, checked out /tmp and /var/tmp, manually monitored the maillog and everything else I can think of, but the "Sent" mail count is still ripping its way through the roof. Now it is sitting at 11,110 in only 1 day.

I don't believe mod_security is installed - I'll check that out, thanks.

If you can offer any other tips that'd be great.

Luke

 

[faris]

Quite franksly I don't understand why it is so difficult to trace the source of problems like this. It can be very frustrating.

Have a serach for "nobody spammer" in these forums for a script that may or may not help.

But first doe a "tail -f /usr/local/psa/var/log/maillog" to view a continuous stream of the log messages for email. You may be able to spot what's happening from there. It can be difficult to interperate what's going on but it might give you a clue.

What we are trying to figure out is whether the emails are being generated on the server, or are coming from the outside somehow.

Faris.

 

[lpittman]

Hey faris,

Finally following up - I had to leave town for a couple weeks.

Since I've left the outgoing email has jumped up to well over 300,000 .. which is worrying me.

I'll check out the modsec logs (and install it if it isn't installed yet) and also tail the log file some more ... but the tail this morning didn't show anything, although, I can't say I know exactly what I am looking for.

Luke

 

[lars.jochimsen]

Hi Luke, hi Farris,

I installed isoqlog as well recently. I was as well caught by surprise, when I saw the immense number of email that was reportet to have been delivered.

isoqlog is telling in the general section:

23 Jul
sent: 18601
received: 7215
summary: 25816

When I enter the details with the top 10/20/100-sections, the top 10 receivers got emails:

01: 1656
02: 191
03: 139
04: 135
05: 128
06: 73
07: 57
08: 38
09: 11
10: 8
--------------
SUM: 2436

So there must have been 7215 - 2436 = 4779 Emails received by other domains on the server not making them part of the top 10??

So I searched the logs to find out, who is getting or sending mail.

In the maillog i grep-ed:
grep -c SEARCH maillog.from.that.day
with SEARCH as:
- starting delivery
- to local
- to remote
getting counts:
- 10190
- 4178
- 6012
getting me to the point:
the grep-ed numbers seem to be the right one, and they match my experience. the grep -c for `identified spam` was 2924 which also seems to be the correct ratio of spam-detection for incoming mails.

So the question arising is:
- are there Mails, that pass by the log? If yes and the server is hacked, why should isoqlog count them, when isoqlog is based on that log
- what exactly is isoqlog counting?? Here seems to be the answer. Of course there is always mail-traffic generated by root@... or anonymous@... (e.g. php-mailers) - so not listet per domain. But you can grep them as well and they as well generate a mail leaving a `to local` or `to remote` in the logs. And also the queue is generating these log-entries when repetetive attempts to mail out occur.

I think this is a bug in isoqmail. Convince me of the opposite - grep your logs and compare.

Tell, what you find out - of course I am not 100% sure and open to be told I'm wrong.

Lars

 

[faris]

I think you are right about the isoqlog possibly being buggy, or possibly it isn't counting things correctly -- I don't know how it works. Is it reading the logs from the correct places?

But I *think* that if there's a PHP script that has its own smtp engine running on your server then emails sent by this would not appear in any log at all. I expect the same thing would be possible in Perl. Please don't assume I'm correct. Also even if I am please don't assume this is what's happening to you -- I very very much doubt it. I think it is just isoqlog not counting things quite correctly.

Faris.