SPAM Problem

[weelk]

I have been working on this problem since about a month with poor results. Going through logs up and down with poor results. I'm really loosing my patience with this issue and my research got me nowhere so far. I'm starting this thread with hope that someone could point me in the right direction. So thanks in advance:

System:
VPS Ubuntu 10.04.4 LTS
Plesk 12.0.18 Update #28
Postfix (relay with authorization)
Dovecot
fail2ban, mod_security

Problem:

• I have email accounts that are sending spam on two domains. I can see this activity in outgoing mail control.
  
• There are three email address that are constantly trying to send emails over limit (15 p/h).
• I have checked with my clients and they don't send this many emails

Action taken so far:

• There is no login fails or suspicious ip in the logs (as far as I'm concerned)
• Logged php headers (if this could originate from website scripts) but it only got me to particular domain's /httpdocs and don't know how to pinpoint the script if any
• Changed passwords on email accounts but this doesn't seem to have any effect on number of emails sent
• Checked FTP logs - nothing
• zgrep 'sasl_method=LOGIN' /usr/local/psa/var/log/maillog* | awk '{print $9}' | sort | uniq -c | sort -nr
  shows like increased number of logins in relation to other accounts

Additional info:

I'm getting brute force attacks from some IPs unfortunately they dont get jailed by fail2ban for some reason, so they keep going.

example from mail.info

Dec 11 05:48:49 My-server plesk_saslauthd[6114]: failed mail authenticatication attempt for user '[email protected]' (password len=7)
Dec 11 05:48:49 My-server postfix/smtpd[6111]: warning: unknown[198.251.79.20]: SASL LOGIN authentication failed: authentication failure
Dec 11 05:48:49 My-server postfix/smtpd[6111]: lost connection after AUTH from unknown[198.251.79.20]
Dec 11 05:48:49 My-server postfix/smtpd[6111]: disconnect from unknown[198.251.79.20]
Dec 11 05:48:49 My-server postfix/smtpd[6111]: connect from unknown[198.251.79.20]

I understand that my post might be lacking some crucial info, but to be honest I'm not sure what else would be needed...

 

[UFHH01]

Hi weelk,

if you experience issues by non-blocking fail2ban intruders, consider posting the fail2ban - log and it's corresponding fail2ban - configurations ( jails ) for mail - filters and actions. Be ware that you might set up the allowed attempts to a lower level and consider using an additional "recidive" - jail, to ban repeat offenders for a longer time.

 

[pleskpanel]

On the mail side if you are changing email passwords for the affected accounts (without even providing the new passwords to your customer) and the message flow continues without stopping it could very well be a script generating the messages (without any authentication but using email addresses at that domain as a return path or as a purported email recipient). While you could also make the case that once the messages being sent after a password change were in fact queued and therefore would be sent regardless of your change, given some of the behavior you described it sounds like you've got an errant script that is being abused in some way. You might be able to test this theory but temporarily moving the entire contents of a low-traffic site generating the spam to a location below the httpdocs directory, restarting Apache, and seeing if it continues to pump out spam in which case it is is unlikely to be the site files and points back to the email account tangent.

 

[weelk]

Thank you for advise guys.Tried to upload the log file to my post as zip and txt format, however I'm getting an error:

There was a problem uploading your file.

So uploaded here https://dl.dropboxusercontent.com/u/1504470/fail2ban.txt

Here is some iptables errors when editing fail2ban jails

2014-12-10 18:28:14,021 fail2ban.actions.action[646]: ERROR   iptables -D INPUT -p tcp -m multiport --dports smtp,smtps,submission -j fail2ban-plesk-postfix
iptables -F fail2ban-plesk-postfix
iptables -X fail2ban-plesk-postfix returned 100
2014-12-10 18:28:14,024 fail2ban.jail   [646]: INFO    Jail 'plesk-postfix' stopped
2014-12-10 18:28:14,120 fail2ban.server [646]: INFO    Changed logging target to /var/log/fail2ban.log for Fail2ban v0.8.13
2014-12-10 18:28:14,121 fail2ban.jail   [646]: INFO    Creating new jail 'plesk-postfix'
2014-12-10 18:28:14,138 fail2ban.jail   [646]: INFO    Jail 'plesk-postfix' uses pyinotify
2014-12-10 18:28:14,220 fail2ban.jail   [646]: INFO    Initiated 'pyinotify' backend
2014-12-10 18:28:14,231 fail2ban.filter [646]: INFO    Added logfile = /var/log/maillog
2014-12-10 18:28:14,233 fail2ban.filter [646]: INFO    Set maxRetry = 3
2014-12-10 18:28:14,234 fail2ban.filter [646]: INFO    Set findtime = 600
2014-12-10 18:28:14,235 fail2ban.actions[646]: INFO    Set banTime = 600
2014-12-10 18:28:14,246 fail2ban.jail   [646]: INFO    Jail 'plesk-postfix' started

In regards to moving files temporary it is rather not possible due to traffic on those websites. Maybe I'll give it a go some time late night/morning. I seriously doubt that emails come from email accounts or infected endpoints. Changed passwords very likely excludes this option. I was hoping there is some relatively simple way to pinpoint malicious scripts.

Both of websites have wordpress installations and both are up to date and most loopholes covered. As you can see I'm not a server guy but I'm a a web developer on steep learning curve. I'm managing my VPS' myself.

 

[UFHH01]

Hi weelk,

please include your "/etc/fail2ban/jail.conf" and the depending files "/etc/fail2ban/jail.local" and "/etc/fail2ban/jail.d/plesk.conf"

If you don't like the investigations to solve misconfigurations ( it looks like it at the moment ), you could as well consider to re-install the Fail2Ban extension with the commands:

/usr/local/psa/admin/bin/autoinstaller --select-product-id plesk --select-release-current --remove-component fail2ban
mv /etc/fail2ban /etc/fail2ban.backup
/usr/local/psa/admin/bin/autoinstaller --select-product-id plesk --select-release-current --install-component fail2ban

... and activate the desired jails again over the Plesk Control Panel afterwards.​

 

[weelk]

Thank you UFHH01. I think it is better to investigate the problem so I learn the lesson for the future. In case something similar takes place.

Here are the files:
https://dl.dropboxusercontent.com/u/1504470/jail.conf.txt
https://dl.dropboxusercontent.com/u/1504470/jail.local.txt
https://dl.dropboxusercontent.com/u/1504470/plesk.conf.txt

 

[UFHH01]

Hi weelk,

at "jail.conf", please change "ignoreip = 127.0.0.1/8" to "ignoreip = 127.0.0.1/8 XXX.XXX.XXX.XXX" , where you please replace XXX... with your server - IP(s) and make sure, each IP is separated with a space. You could be blocking your own server, when this definition is missing.

at "jail.conf", please change "backend = auto" to "backend = gamin". On some system configuration, gamin or polling work better than "pyinotify"
If gamin is not installed, please use "apt-get install gamin" to install it.
Try out as well "polling" and judge after a few days, which configuration suits your system best.

Your "jail.local" contains additional modifications to the standard, pre-configured jails from Plesk. Did you just "play around" or did you modify the jails, because you have made a good experience with these settings? Please consider switching back to the standard configuration, which you find at "/etc/fail2ban/jail.d/plesk.conf" and only modify the enabled - part from false to true, in order to investigate issues and failures, when you copy the standard - Plesk - configuration from plesk.conf to jail.local. Modify the jails only if all works as expected and then as well only ONE BY ONE, so you can investigate possible failures far better.


After all these steps... please restart fail2ban with the command "service fail2ban restart" and report again issues/failures or maybe success informations. ^^

 

[Linulex]

just a shot in the dark: are there drupal sites on the server? about a month ago there was a very serious security issue in drupal and your problems started about a month ago. updating the drupal wasnt enough to fix it. You had to manually clean all the infected files or replace the complete site with a backup from before the infection and then update the drupal.

You can run this from the commandline to find infected files, if it was the drupal hack.

find /var/www/vhosts/ -exec grep -liI "PCT4BA6ODSE" {} \; > /root/PCT4BA6ODSE.txt

the file PCT4BA6ODSE.txt will have a list, if any, infected files

more on finding an cleaning here.

http://allabouttodd.com/drupal/hacked-PCT4BA6ODSE

regards
Jan

 

[weelk]

Thank you for your help UFHH01. I really appreciate it

1. changed "backend = auto" to "backend = gamin"
2. Installed gamin
3. Copied over default config

SO far so good. log looks much better now. I will check back later to see if everything is fine.


@Linulex thanks for suggestion but no drupals on this server. Also checked existing scripts and couldn't find anything suspicious.