Resolved Spam problem

[nubevps]

Hello,

Since a week ago I have a Plesk 12.5 Linux server using Postfix with spam problem similar to this thread:

https://talk.plesk.com/threads/spam-from-hole-in-plesk.290730/page-3

no spam logged in /var/log/maillog, neither from php scripts monitored on /var/log/phpmaillog, neither on Plesk outgoing queue.

I also followed instructions of https://support.plesk.com/hc/en-us/articles/213914405 but no spam is recorded.

Found apache cron on /var/spool/cron/ created recently with this content:

*/10 * * * * /var/tmp/UJUDLy >/dev/null 2>&1

file /var/tmp/UJUDLy is missing now, it must be the spam source.

when spam process is running it looks like this:

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1734 apache 20 0 43540 6192 832 S 6,2 0,1 0:00.49 exim
1881 apache 20 0 44856 8240 1440 S 6,2 0,1 14:53.25 exim

and here details of process:

[root@linux ~]# lsof -p 1881
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
exim 1881 apache cwd DIR 8,3 4096 2 /
exim 1881 apache rtd DIR 8,3 4096 2 /
exim 1881 apache txt REG 8,3 11400 769150 /usr/bin/perl
exim 1881 apache mem REG 8,3 28272 834386 /usr/lib64/perl5/auto/File/Glob/Glob.so
exim 1881 apache mem REG 8,3 86872 831588 /usr/lib64/perl5/auto/POSIX/POSIX.so
exim 1881 apache mem REG 8,3 19504 831553 /usr/lib64/perl5/auto/Fcntl/Fcntl.so
exim 1881 apache mem REG 8,3 44520 831342 /usr/lib64/perl5/vendor_perl/auto/Socket/Socket.so
exim 1881 apache mem REG 8,3 19800 831569 /usr/lib64/perl5/auto/IO/IO.so
exim 1881 apache mem REG 8,3 11376 766893 /usr/lib64/libfreebl3.so
exim 1881 apache mem REG 8,3 2112384 766114 /usr/lib64/libc-2.17.so
exim 1881 apache mem REG 8,3 142304 766140 /usr/lib64/libpthread-2.17.so
exim 1881 apache mem REG 8,3 14608 769425 /usr/lib64/libutil-2.17.so
exim 1881 apache mem REG 8,3 40816 767123 /usr/lib64/libcrypt-2.17.so
exim 1881 apache mem REG 8,3 1141560 770824 /usr/lib64/libm-2.17.so
exim 1881 apache mem REG 8,3 19520 770817 /usr/lib64/libdl-2.17.so
exim 1881 apache mem REG 8,3 113328 770829 /usr/lib64/libnsl-2.17.so
exim 1881 apache mem REG 8,3 110808 770833 /usr/lib64/libresolv-2.17.so
exim 1881 apache mem REG 8,3 1643144 834384 /usr/lib64/perl5/CORE/libperl.so
exim 1881 apache mem REG 8,3 28120 770814 /usr/lib/libsafe.so.2.0.16
exim 1881 apache mem REG 8,3 164440 771212 /usr/lib64/ld-2.17.so
exim 1881 apache 0r CHR 1,3 0t0 1028 /dev/null
exim 1881 apache 1w CHR 1,3 0t0 1028 /dev/null
exim 1881 apache 2w CHR 1,3 0t0 1028 /dev/null
exim 1881 apache 4u IPv4 277247201 0t0 TCP myserver.hostname:30900->smtp-in.orange.fr:smtp (ESTABLISHED)
exim 1881 apache 5u IPv4 277244485 0t0 UDP *:52452
exim 1881 apache 6u IPv4 277247203 0t0 TCP myserver.hostname:30902->smtp-in.orange.fr:smtp (ESTABLISHED)
exim 1881 apache 7u IPv4 277246087 0t0 UDP *:53609
exim 1881 apache 9u IPv4 277247202 0t0 TCP myserver.hostname:30901->smtp-in.orange.fr:smtp (ESTABLISHED)
exim 1881 apache 10u IPv4 277248023 0t0 UDP *:51456
exim 1881 apache 11u IPv4 277242683 0t0 UDP *:40147
exim 1881 apache 12u IPv4 277247204 0t0 TCP myserver.hostname:30903->smtp-in.orange.fr:smtp (ESTABLISHED)
exim 1881 apache 13u IPv4 277246243 0t0 UDP *:61612
exim 1881 apache 14u IPv4 277247200 0t0 TCP myserver.hostname:30899->smtp-in.orange.fr:smtp (ESTABLISHED)
exim 1881 apache 15u IPv4 277242717 0t0 UDP *:56766
exim 1881 apache 16u IPv4 277247205 0t0 TCP myserver.hostname:30904->smtp-in.orange.fr:smtp (ESTABLISHED)
exim 1881 apache 17u IPv4 277247198 0t0 TCP myserver.hostname:30897->smtp-in.orange.fr:smtp (ESTABLISHED)
exim 1881 apache 18u IPv4 277244500 0t0 TCP myserver.hostname:63439->mtain-a-mtc-c.mx.aol.com:smtp (SYN_SENT)
exim 1881 apache 20u IPv4 277247199 0t0 TCP myserver.hostname:30898->smtp-in.orange.fr:smtp (ESTABLISHED)
exim 1881 apache 21u IPv4 277247214 0t0 UDP *:31989
exim 1881 apache 26u IPv4 277245558 0t0 TCP myserver.hostname:38309->al-ip4-mx-vip1.prodigy.net:smtp (ESTABLISHED)
exim 1881 apache 29u IPv4 277245570 0t0 TCP myserver.hostname:36741->mtain-b-mtc-b.mx.aol.com:smtp (ESTABLISHED)
exim 1881 apache 30u IPv4 277248062 0t0 UDP *:50160
exim 1881 apache 32u IPv4 277246328 0t0 TCP myserver.hostname:30984->smtp-in.orange.fr:smtp (ESTABLISHED)
exim 1881 apache 34u IPv4 277246336 0t0 TCP myserver.hostname:64853->ff-ip4-mx-vip1.prodigy.net:smtp (ESTABLISHED)
exim 1881 apache 37u IPv4 277242858 0t0 TCP myserver.hostname:49294->mx1.hotmail.com:smtp (SYN_SENT)
exim 1881 apache 38u IPv4 277246342 0t0 TCP myserver.hostname:64859->ff-ip4-mx-vip1.prodigy.net:smtp (ESTABLISHED)
exim 1881 apache 39u unknown /proc/1881/fd/39 (readlink: No such file or directory)
exim 1881 apache 45u IPv4 277246284 0t0 TCP myserver.hostname:38332->al-ip4-mx-vip1.prodigy.net:smtp (ESTABLISHED)
exim 1881 apache 46u IPv4 277246338 0t0 TCP myserver.hostname:35119->mxzhb.bluewin.ch:smtp (ESTABLISHED)
exim 1881 apache 49u IPv4 277245970 0t0 TCP myserver.hostname:49370->mx1.hotmail.com:smtp (ESTABLISHED)
exim 1881 apache 50u IPv4 277246288 0t0 TCP myserver.hostname:44628->extmail.optusnet.com.au:smtp (ESTABLISHED)
exim 1881 apache 56u IPv4 277246297 0t0 TCP myserver.hostname:50161->mx1.hotmail.com:smtp (ESTABLISHED)
exim 1881 apache 61u IPv4 277247233 0t0 TCP myserver.hostname:31003->smtp-in.orange.fr:smtp (ESTABLISHED)
exim 1881 apache 67u IPv4 277246330 0t0 TCP myserver.hostname:30986->smtp-in.orange.fr:smtp (ESTABLISHED)
exim 1881 apache 69u IPv4 277246324 0t0 TCP myserver.hostname:30980->smtp-in.orange.fr:smtp (ESTABLISHED)
exim 1881 apache 73u IPv4 277246329 0t0 TCP myserver.hostname:30985->smtp-in.orange.fr:smtp (ESTABLISHED)
exim 1881 apache 74u IPv4 277246326 0t0 TCP myserver.hostname:30982->smtp-in.orange.fr:smtp (ESTABLISHED)
exim 1881 apache 75u IPv4 277245980 0t0 TCP myserver.hostname:49390->mx1.hotmail.com:smtp (ESTABLISHED)
exim 1881 apache 77u IPv4 277246341 0t0 TCP myserver.hostname:64858->ff-ip4-mx-vip1.prodigy.net:smtp (ESTABLISHED)
exim 1881 apache 80u IPv4 277246001 0t0 TCP myserver.hostname:49536->mx1.hotmail.com:smtp (SYN_SENT)
exim 1881 apache 84u IPv4 277246321 0t0 TCP myserver.hostname:61822->etb-1.mail.tiscali.it:smtp (ESTABLISHED)
exim 1881 apache 85u IPv4 277246327 0t0 TCP myserver.hostname:61826->etb-1.mail.tiscali.it:smtp (ESTABLISHED)
exim 1881 apache 86u IPv4 277246331 0t0 TCP myserver.hostname:30987->smtp-in.orange.fr:smtp (ESTABLISHED)
exim 1881 apache 88u IPv4 277246334 0t0 TCP myserver.hostname:30990->smtp-in.orange.fr:smtp (ESTABLISHED)
exim 1881 apache 242w FIFO 0,8 0t0 269682974 pipe
exim 1881 apache 243r FIFO 0,8 0t0 269682985 pipe
exim 1881 apache 244w FIFO 0,8 0t0 269682985 pipe
exim 1881 apache 245r FIFO 0,8 0t0 269682986 pipe
exim 1881 apache 246w FIFO 0,8 0t0 269682986 pipe
exim 1881 apache 247w REG 8,2 9480894 4806977 /var/log/httpd/mod_jk.log
exim 1881 apache 248u REG 8,2 1024 4800845 /var/log/httpd/jk-runtime-status.22449 (deleted)
exim 1881 apache 249u REG 8,2 1 4801066 /var/log/httpd/jk-runtime-status.22449.lock (deleted)

*replaced real server for myserver.hostname.

Passed Maldet (no threats found) and ClamAV (19 infected files) to /var/www/vhosts/.

Any suggestion to stop this?, I just added apache to /etc/cron.deny and I´m waiting spam to reappear.

 

[Bitpalast]

The issue is caused by a temporary file that is installed by a malicious script from one website into the temp directory, then executed by a crontab job. This file starts an additional mail server like "exim", so that malware can use that mailserver to send spam from your server. A crontab job is used for it, because this circumvents execution permission limits. As that instance of the additional mail server does not log anything to the default log files, you cannot see anything in your logs about it.

In order to avoid this issue you must make sure that files in the temp directory or partition cannot be executed. See the section "restricting script execution" in this: https://docs.plesk.com/en-US/12.5/advanced-administration-guide-linux/enhancing-security.68755/

 

[nubevps]

thanks for information Peter.

Do you know any way to find the malicious script?, I passed Maldet and ClamAV to full /var/www/vhosts/*.

 

[Bitpalast]

No. It may not be possible to find it at all. Normally, you won't find the script, but only a script that downloads another script or creates another script that does something that does something ... It is not even clear whether the initial script is located on the server. It can equally well be located on a user's computer and act as a user by posting and getting data through port 80 or 443, filling in the "planned jobs" fields of cron, saving them and so on.

 

[nubevps]

Ok I see Peter,

What I don´t understand is how this mailicious code can open ports bypassing current APF firewall configuration, this is yesterday´s warning from LSM socket/port alert monitor:

Following is a summary of new Internet Server Sockets:
> tcp 0 0 0.0.0.0:43210 0.0.0.0:* LISTEN 30609/init

opened port 43210 that is closed in firewall.

 

[nubevps]

forgot to tell you that I just found spam script on tmp folder of ClamAV antivirus:

/tmp/systemd-private-5eee1220f76546f3b8aa8bf4847dcd9a-clam-freshclam.service-bA1bUi

is this normal?.

 

[Bitpalast]

Maybe your server security was compromised so that a malware was installed beyond Plesk? After all, this case is not an issue of the Plesk software, but a security issue of the operating system. You might need expert support on your Linux installation or maybe need to post the issue to a forum that specializes on such cases (no, I cannot recommend any).

 

[nubevps]

well rkhunter and chkrootkit don´t show anything infected but you can´t trust in anything in this business :-(

 

[nubevps]

I set all domains to use fastcgi as apache handler with this command:

for i in `mysql -uadmin -p\`cat /etc/psa/.psa.shadow\` psa -Ns -e "select name from domains"`; do /usr/local/psa/bin/domain -u $i -php_handler_type fastcgi; done

is there any way to stop apache running processes?, just the php-cgi process of each domain alone.

 

[nubevps]

Hi Peter,

I was able to catch the spammer and eliminate it , very long history...

Happy new year to all forum members.

Regards.
Victor

 

[jargonar]

Victor can you tell the comunity how u solved this problem?

Thnx. Gustavo

 

[nubevps]

Hi,

Probably you have an infected CMS in your server (Joomla, Wordpress...) where attacker upladed spam script.

First you need to install in your server LSM (Linux Socket Monitor): http://rfxnetworks.com/downloads/lsm-current.tar.gz

This will warn you when a new socket is opened in your server, that means spammer has executed his script.

Usually these scripts are php files so you need to catch it on execution. In order to do this you must be sure all your domains use fastcgi as Apache handler (see my reply above) so all apache process are executed under his user.

When you get alert email from LSM then you monitor your server with top command and check for apache running processes. You will see a process running for sometime, that should be the spammer, check the user of that process and inspect access_log of associated domain. Check log in the time of the attack, you should see an entry similar to this:

X.X.X.X - - [29/Dec/2016:21:42:51 +0100] "GET /modules/mod_fxprev/libraries/httpd.pl HTTP/1.0" 500 709 "-" "-"

There you have the spammer script: httpd.pl

Delete script and secure website, in my case it was a Joomla with insecure /mod_fxprev module.

Also check the users cron (/var/spool/cron/) where attacker normally schedule the script´s execution. Default content of Plesk user cron is the following:

SHELL=/usr/local/psa/bin/chrootsh
MAILTO=""

with file size 44k. Check the list of users with a different cron size and inspect them individually. Infected user should have something in his cron linked with a file in your /tmp folder. Delete the entry and save cron.

*Good luck with the hunt!