Question Virus or bad script or none of the above?

[Gareth Westwood]

Hi All,

First time posting here but long time user of plesk (15 years probably!!!)

CentOS 6.10
Plesk Onyx 17.5.3
Mainly hosting low traffic websites and some email (~60 subscriptions)

I was recently contacted by my VPS provider to say that IP address allocated to my server have been performing port scans and that they have been contacted and asked to investigate.

I was not totally sure where to start but have gone through a few steps to try and isolate the issue. I'm wondering if anyone here can offer any further advice.

last, lastb and lastlog all look ok. So I don't think anyone has compromised a "proper" system account. who only shows my current login

find / -perm -4000 -perm -2000 only shows one file, /usr/local/psa/handlers/hooks/check-quota . Not sure if that is supposed to have those permissions or not.

I ran top to see if anything was obviously wrong there. I couldn't see anything but did note two things. Firstly that load average seems to be between 2 and 10! which seems really high the second being that the CPU %us (from line 3 of top) is significantly higher than the top 10 ish process %CPU values. I'm not sure what I was expecting to see here but I guess I thought that whatever is increasing my load average would be showing up here.

netstat -plunt is also looking fine there are a few names I don't recognize (monit and master being the main two) but again nothing is jumping out at me as being wrong.

I ran find /var/www/vhosts -mtime -30 and found a couple of domains with "suspect" recently modified files so have disabled those two sites for now.

Last but not least I have added some firewall rules to block inbound and outbound on port 22 other than from my office IP address and one other (backup) site. I am running watch iptables -L OUTPUT -v and I am seeing some packets hit the drop rule so _something_ is trying to dial out.

What would you guys suggest as the next step in working out what is going on here. I guess there may be a way to work out which binary made the dropped packets, then work out which user ran the binary maybe?

Your thoughts would be appreciated!

 

[IgorG]

Try to check your Plesk server with rkhunter - How to inspect a machine with the help of rkhunter?
This article also may be useful - Plesk security best practices

 

[Gareth Westwood]

Hi Igor,

Thanks, I didn't think to check with rkhunter. I will run it in a few mins and see what the results are. I am also in the process of installing maldect + clamd to see if they find anything.

 

[Gareth Westwood]

ok, this machine is running fairly badly. maldect was still running so I just logged on to see how it's looking and the load average was 80+. rkhunter has produced a warning for /bin/mailx and is currently on "additional rootkit checks".

I have however spotted a load of python commands in top that appeared to be sending mail out. They vanished from the screen before I had chance to do anything though. I have also spotted several of the following line;

7289 apache 20 0 11836 2004 1216 S 0.3 0.0 0:00.97 /bin/bash ./start

Which looks suspect. I've got a find / -name start -print running at the moment.

 

[Gareth Westwood]

Would someone be able to check through this crontab listing and let me know if it all looks standard?
15 1 * * 1 /usr/local/psa/libexec/modules/watchdog/cp/pack-sysstats week
0 1 * * 1 /usr/local/psa/libexec/modules/watchdog/cp/secur-check
15 1 * * * /usr/local/psa/libexec/modules/watchdog/cp/pack-sysstats day
MAILTO=""
43 * * * * /usr/local/psa/admin/bin/php -dauto_prepend_file=sdk.php '/usr/local/psa/admin/plib/modules/letsencrypt/scripts/keep-secured.php'
MAILTO="[email protected]"
20 1 * * * /usr/local/psa/libexec/modules/watchdog/cp/clean-events
10 1 * * * /usr/local/psa/libexec/modules/watchdog/cp/clean-sysstats
15 1 1 * * /usr/local/psa/libexec/modules/watchdog/cp/pack-sysstats month
0 1 * * 1 /usr/local/psa/libexec/modules/watchdog/cp/send-report weekly
15 1 1 * * /usr/local/psa/libexec/modules/watchdog/cp/pack-sysstats year
0 3 * * 7 /usr/local/psa/libexec/modules/watchdog/cp/clean-reports
MAILTO=""
0 0 * * * /usr/local/psa/admin/bin/php -dauto_prepend_file=sdk.php '/usr/local/psa/admin/plib/modules/wp-toolkit/scripts/instances-auto-update.php'
MAILTO=""
# 0,10,20,30,40,50 * * * * /usr/local/psa/admin/bin/php -dauto_prepend_file=sdk.php '/usr/local/psa/admin/plib/modules/plesk-mobile/scripts/push_worker.php'
MAILTO=""
49 * * * * /usr/local/psa/admin/bin/php -dauto_prepend_file=sdk.php '/usr/local/psa/admin/plib/modules/wp-toolkit/scripts/maintenance.php'

 

[Gareth Westwood]

Ok, I've got the results from rkhunter and have copied the (I hope) relevant bits from the log below;

[13:05:33] /sbin/ip [ Warning ]
[13:05:36] Warning: The file properties have changed:
[13:05:38] File: /sbin/ip
[13:05:39] Current inode: 38932712 Stored inode: 38932716

[13:14:45] /bin/mailx [ Warning ]
[13:14:46] Warning: The file properties have changed:
[13:14:46] File: /bin/mailx
[13:14:47] Current inode: 38932881 Stored inode: 38932566

[14:22:58] Checking for suspicious shared memory segments [ Warning ]
[14:22:59] Warning: The following suspicious shared memory segments have been found:
[14:22:59] Process: /usr/sbin/httpd PID: 1274 Owner: root
[14:23:01] Process: PID: 14272 Owner: root

[14:23:49] Checking '/etc/xinetd.d/ftp_psa' for enabled services [ Warning ]
[14:23:52] Checking '/etc/xinetd.d/ntalk' for enabled services [ None found ]
[14:23:54] Checking '/etc/xinetd.d/poppassd_psa' for enabled services [ Warning ]
[14:23:57] Checking '/etc/xinetd.d/rsync' for enabled services [ None found ]
[14:23:59] Checking '/etc/xinetd.d/talk' for enabled services [ None found ]
[14:24:01] Checking '/etc/xinetd.d/tcpmux-server' for enabled services [ None found ]
[14:24:03] Checking '/etc/xinetd.d/time-dgram' for enabled services [ None found ]
[14:24:05] Checking '/etc/xinetd.d/time-stream' for enabled services [ None found ]
[14:24:06] Checking for enabled xinetd services [ Warning ]
[14:24:08] Warning: Found enabled xinetd service: /etc/xinetd.d/ftp_psa
[14:24:09] Warning: Found enabled xinetd service: /etc/xinetd.d/poppassd_psa
[14:24:10] Checking for Apache backdoor [ Not found ]
[14:24:11]
[14:24:11] Info: Starting test name 'os_specific'
[14:24:12] Performing Linux specific checks
[14:24:14] Checking loaded kernel modules [ Warning ]
[14:24:16] Warning: No output found from the lsmod command or the /proc/modules file:
[14:24:16] /proc/modules output:
[14:24:17] lsmod output:

[16:41:42] Checking for passwd file changes [ Warning ]
[16:41:42] Warning: User 'xxxxxx' has been removed from the passwd file.
(there were several of these but they were all for domains I have removed whilst tidying things up
[16:41:46] Warning: User 'xxxxxx' has been added to the passwd file.
(two of these one for my new none root user and the other for clam av that I installed yesterday)
[16:41:47] Checking for group file changes [ Warning ]
[16:41:47] Warning: Group 'xxxxxx' has been added to the group file.
(two of these, one for each of the new users)

[16:41:50] Checking if SSH root access is allowed [ Warning ]
[16:41:51] Warning: The SSH and rkhunter configuration options should be the same:
[16:41:51] SSH configuration option 'PermitRootLogin': no
[16:41:51] Rkhunter configuration option 'ALLOW_SSH_ROOT_USER': unset
[16:41:51] Checking if SSH protocol v1 is allowed [ Warning ]
[16:41:52] Warning: The SSH and rkhunter configuration options should be the same:
[16:41:52] SSH configuration option 'Protocol': 2
[16:41:52] Rkhunter configuration option 'ALLOW_SSH_PROT_V1': 2

[16:42:09] Checking /dev for suspicious file types [ Warning ]
[16:42:10] Warning: Suspicious file types found in /dev:
[16:42:10] /dev/.udev/queue.bin: data
[16:42:12] Checking for hidden files and directories [ Warning ]
[16:42:13] Warning: Hidden directory found: /dev/.udev
[16:42:13] Warning: Hidden file found: /usr/share/man/man5/.k5identity.5.gz: gzip compressed data, from Unix, max compression
[16:42:13] Warning: Hidden file found: /usr/share/man/man5/.k5login.5.gz: gzip compressed data, from Unix, max compression
[16:42:14] Warning: Hidden file found: /usr/share/man/man1/..1.gz: gzip compressed data, from Unix, max compression
[16:42:14] Warning: Hidden file found: /usr/bin/.ssh.hmac: ASCII text
[16:42:14] Warning: Hidden file found: /usr/bin/.fipscheck.hmac: ASCII text
[16:42:15] Warning: Hidden file found: /usr/sbin/.sshd.hmac: ASCII text

 

[MarkM]

Check the temp mount and make sure it’s mounted as noexec;

Enhancing Security

It’s possible that you have a script somewhere on the server that is downloading another script to tmp which if so will make the tracking down a million times more difficult

 

[Gareth Westwood]

Thanks Mark,

/tmp is not on a separate partition, can I set a folder noexec? (I'll ask google in a sec). It does look like there are some "interesting" files in there. Would the recomendation be to delete them, move them chown root:root then chmod 400?

I've just installed audit to try and track down what's happening but (bloody typically) I'm struggling to make it start now.

 

[MarkM]

Look at the second part of the link, it has details on creating a new tmp partion on the current filesystem. You could also try a more aggressive move and chmod the Perl binary and zero it out....

 

[Gareth Westwood]

Looks like I don't have the loop stuff in the kernel.

mount: Could not find any loop device. Maybe this kernel does not know
about the loop device? (If so, recompile or `modprobe loop'.)

modprobe loop gives

FATAL: Module loop not found.

That's a bit of a pain. I cleared the suspect stuff out of tmp on Friday and there was stuff back there today. The files are owned by apache which also fits with a script downloading them. Can anyone suggest a way for me to find out which site contains the script that wrote the file out. I presume it will be possible to track it down with maybe via access_log?

I'll work through the rest of the enhancing security doc today, maybe the fast cgi bit will lead to the files being written out as the site user rather than apache?

 

[Bitpalast]

Normally this type of attack utilizes a crontab script to place the files into /tmp and to run them from there, because by crontab it is easier to circumvent subscription security measures. My suggestion is to first go through all cron entries ("scheduled tasks") and remove suspicious tasks from there.

You MUST definitely follow Mark's suggestion (link to Plesk doc) that describes how to create a non-executable /tmp directory. It is not enough to change permissions by chmod. When you do not make this change, your system remains vulnerable.

 

[Gareth Westwood]

Hey Peter,

I went through the crontabs for all users and dropped a copy of the root one in reply 5 above. Difficult to know what is suspicious without having a clean install to check against. Would anyone be good enough to have a quick glance at that and see if it looks about right.

Waiting for a response back on a centos forum about the total lack of loadable kernel modules on my system (which is also affecting my attempts to improve security with iptables). As soon as I can load a loopback file system I'll sort that out (the rest is done, just can't mount it).

Thanks for the ongoing help guys.

 

[Gareth Westwood]

and for the record, if anyone else has an issue mounting a loopback filesystem / loading modules...

It turns out that my server is an openvz container and I need to talk to my hosting provider and see if they can give me access to the stuff I'm missing. Will see what they say.

 

[Bitpalast]

The Crontab list that you have presented in this case is not your full crontab. It is only the part that refers to root or admin. A good way to see potential other entries is to open the "scheduled tasks" on administrator level in the GUI. It shows all crontab entries - the system entries and the user entries.

 

[Gareth Westwood]

Thanks Peter I'll check that out as well.

For the record I did run a command to iterate through /etc/passwd and list the crontab for every user. The other users did have a couple of things listed (mostly default shell settings) but I cleared them all out.

for user in $(cut -f1 -d: /etc/passwd); do echo $user; crontab -u $user -l; done​

 

[Gareth Westwood]

The scheduled tasks in the plesk gui shows the same as I copied and pasted above. It all looks ok to me (unless something has managed to write a script to /var/local/psa.

Any other ideas?