1. Please take a little time for this simple survey! Thank you for participating!
    Dismiss Notice
  2. Dear Pleskians, please read this carefully! New attachments and other rules Thank you!
    Dismiss Notice
  3. Dear Pleskians, I really hope that you will share your opinion in this Special topic for chatter about Plesk in the Clouds. Thank you!
    Dismiss Notice

Server compromised?

Discussion in 'Plesk for Linux - 8.x and Older' started by lpittman, May 13, 2007.

  1. lpittman

    lpittman Guest

    Hey team ...

    I installed isoqlog to see what the usage was like on my server and was alarmed to see 6500+ "Sent" emails after scanner only todays log.

    I run 'isologd' every 10 minutes or so just to see and the Sent emails increases by a nearly 100 every time. This mail server _does not_ send out that many legitimate outgoing emails.

    So, I have two thoughts:

    1. I do have qgreylist running - so, technically if I am receiving a lot of spam, this would be sending back a lot of emails, right?

    2. The server has been compromised.

    I've checked /tmp and /var/tmp and see nothing out of the ordinary. I have run chkrootkit and rkhunter and neither reports anything strange.

    Can anyone offer any advice here?

  2. faris

    faris Guest

    qgreylist does not send out any emails to incoming spam. The "come back later" type "message" is actually part of the handshake if you like. So it won't generate any messages at all when it is graylisting.

    So yes, someone could have found a way to compromise a script on the server, or could have installed a spamming script. But it could also be one of your customers who is accidentally doing this.

    For example one of our customers had a mailserver at their side that was compromised. Unfortunately it was set to use the smtp service on their account on our servers. So the spam went out via us rather than them (though we found the problem and dealt with it before too much harm had been done).

    It could also be one of your customers is being spammed to high heaven, and has their account set to reply to emails sent to non-existant users or something like that.

    As a side note I have noticed that spammers have been particularly active over this weekend. This is unusual - normally the peak is on Thursdays in my experience.

    Anyway, I don't know what else to suggest you really...if you have mod_security installed take a look through the log to see if any particular site has been probed a lot, or if any particular IP has been probing, then go through the website logs to find a match maybe).

    Also do the usual rkhunter type stuff to make sure nothing serious has happened.

  3. lpittman

    lpittman Guest


    Thanks for the reply.

    I should have realized qgreylist doesn't send emails back - good point.

    I've run rkhunter and chkrootkit, checked out /tmp and /var/tmp, manually monitored the maillog and everything else I can think of, but the "Sent" mail count is still ripping its way through the roof. Now it is sitting at 11,110 in only 1 day.

    I don't believe mod_security is installed - I'll check that out, thanks.

    If you can offer any other tips that'd be great.

  4. faris

    faris Guest

    Quite franksly I don't understand why it is so difficult to trace the source of problems like this. It can be very frustrating.

    Have a serach for "nobody spammer" in these forums for a script that may or may not help.

    But first doe a "tail -f /usr/local/psa/var/log/maillog" to view a continuous stream of the log messages for email. You may be able to spot what's happening from there. It can be difficult to interperate what's going on but it might give you a clue.

    What we are trying to figure out is whether the emails are being generated on the server, or are coming from the outside somehow.

  5. lpittman

    lpittman Guest

    Hey faris,

    Finally following up - I had to leave town for a couple weeks.

    Since I've left the outgoing email has jumped up to well over 300,000 .. which is worrying me.

    I'll check out the modsec logs (and install it if it isn't installed yet) and also tail the log file some more ... but the tail this morning didn't show anything, although, I can't say I know exactly what I am looking for.

  6. lars.jochimsen

    lars.jochimsen Guest

    Hi Luke, hi Farris,

    I installed isoqlog as well recently. I was as well caught by surprise, when I saw the immense number of email that was reportet to have been delivered.

    isoqlog is telling in the general section:

    23 Jul
    sent: 18601
    received: 7215
    summary: 25816

    When I enter the details with the top 10/20/100-sections, the top 10 receivers got emails:

    01: 1656
    02: 191
    03: 139
    04: 135
    05: 128
    06: 73
    07: 57
    08: 38
    09: 11
    10: 8
    SUM: 2436

    So there must have been 7215 - 2436 = 4779 Emails received by other domains on the server not making them part of the top 10??

    So I searched the logs to find out, who is getting or sending mail.

    In the maillog i grep-ed:
    grep -c SEARCH maillog.from.that.day
    with SEARCH as:
    - starting delivery
    - to local
    - to remote
    getting counts:
    - 10190
    - 4178
    - 6012
    getting me to the point:
    the grep-ed numbers seem to be the right one, and they match my experience. the grep -c for `identified spam` was 2924 which also seems to be the correct ratio of spam-detection for incoming mails.

    So the question arising is:
    - are there Mails, that pass by the log? If yes and the server is hacked, why should isoqlog count them, when isoqlog is based on that log
    - what exactly is isoqlog counting?? Here seems to be the answer. Of course there is always mail-traffic generated by root@... or anonymous@... (e.g. php-mailers) - so not listet per domain. But you can grep them as well and they as well generate a mail leaving a `to local` or `to remote` in the logs. And also the queue is generating these log-entries when repetetive attempts to mail out occur.

    I think this is a bug in isoqmail. Convince me of the opposite - grep your logs and compare.

    Tell, what you find out - of course I am not 100% sure and open to be told I'm wrong.

  7. faris

    faris Guest

    I think you are right about the isoqlog possibly being buggy, or possibly it isn't counting things correctly -- I don't know how it works. Is it reading the logs from the correct places?

    But I *think* that if there's a PHP script that has its own smtp engine running on your server then emails sent by this would not appear in any log at all. I expect the same thing would be possible in Perl. Please don't assume I'm correct. Also even if I am please don't assume this is what's happening to you -- I very very much doubt it. I think it is just isoqlog not counting things quite correctly.