• If you are still using CentOS 7.9, it's time to convert to Alma 8 with the free centos2alma tool by Plesk or Plesk Migrator. Please let us know your experiences or concerns in this thread:
    CentOS2Alma discussion

Possibly Hacked - High Traffic and Httpd

R

ryanz

Basic Pleskian
Hi,

I'm not sure what the cause is or if it's coincidence but our server stared behaving strangely just a few days after the upgrade to 8.4.0 on Cent 0S 5-64.

The outgoing traffic spikes to 50-80Mbs for an hour or so then goes back to normal for a while then just spikes back up again.

During this time TOP shows 2 httpd processes running at 99% CPU and iftop shows high outgoing traffic to a number of domains and ips. Ntop shows outgoing traffic to be UDP but we cannot find the source.

We have APF, BFD, Mod_Security, chkrootkit, rkhunter running with a secured /tmp and can't find anything obvious.

Has anyone had this experience and what are the steps to follow to try and solve this?
 
That is pretty suspicious, you could try grabbing a sniffer dump the next time it happens. Another good tool is www.mod-top.org, which will let you look into what apache is doing.

A forensic first step would be to run rpm -Va on the system, and look into all the changes against the base rpm settings. There will be a lot, its not the kind of thing where there are any shortcuts I'm afraid. Second, any standard rootkit will more or less be able to hide from that anyway.
 
This is a kind of "leap in the dark" but if you aren't using the DNS service you can try to disable it. I'm experiencing the same problem since Sunday night and after a lot of tests I have found that disabling the DNS service in Plesk, the traffic just dropped to the normal amount.
 
All that means is you're disabling DNS lookups for whatever is going on. It would certainly break anything nefarious going on that was DNS based, at least until the badguys notice and change it back :p
 
Thanks for the suggestions so far.

I did a test and closed all UDP ports in APF, since then there seems to be no problem with the server running at normal traffic for about 10 hours.

Any suggestions?

Scott, what should I be looking for with rpm -Va? Here are some details:
.M...... /usr/bin/curl
.......T c /etc/libuser.conf
.......T c /etc/mail/sendmail.cf
S.5....T c /var/log/mail/statistics
S.5....T c /etc/ssh/sshd_config
.......T d /usr/share/man/man1/afs5log.1.gz
.......T d /usr/share/man/man5/pam_krb5.5.gz
.......T d /usr/share/man/man8/pam_krb5.8.gz
.......T d /usr/share/man/man8/pam_krb5_storetmp.8.gz
.M...... /usr/bin/wget
..5....T c /etc/pki/nssdb/secmod.db
.......T /usr/share/X11/XErrorDB
S.5....T c /etc/php.ini
.......T /usr/include/pcap-bpf.h
.......T /usr/include/pcap-namedb.h
.......T /usr/include/pcap.h
prelink: /usr/bin/pk11install: at least one of file's dependencies has changed since prelinking
S.?..... /usr/bin/pk11install
 
I installed mod_top and ran it for a while to see the processes, then I opened the UDP ports in APF and after a few minutes there was high httpd/apache load with very high traffic.

TOP showed two httpd processes running at 99% cpu each but mod_top showed only one running at 28% cpu.

If I restart or stop httpd while the load and traffic is high everything returns to normal and then simply goes up again after a few minutes.

We get problems when we open these ports:
IG_UDP_CPORTS="37,53,161,873"
EG_UDP_CPORTS="53,873"
 
By the way, I installed Plesk using the Autoinstaller and also did the upgrade to 8.4.0 using the auto updater in the CP.

According to yum check-update these are the only packages that are not updated and I'm worried they might break something or cause problems with future auto updates.

httpd.x86_64 2.2.3-11.el5_1.centos. base
kbd.x86_64 1.12-20.el5 base
kernel.x86_64 2.6.18-92.1.6.el5 updates
kernel-devel.x86_64 2.6.18-92.1.6.el5 updates
kernel-headers.x86_64 2.6.18-92.1.6.el5 updates
libhugetlbfs.x86_64 1.2-5.el5 base
mod_ssl.x86_64 1:2.2.3-11.el5_1.cento base
mysql.i386 5.0.45-7.el5 base
mysql.x86_64 5.0.58-1.el5.art atomic
mysql-server.x86_64 5.0.58-1.el5.art atomic
php.x86_64 5.2.6-1.el5.art atomic
php-cli.x86_64 5.2.6-1.el5.art atomic
php-common.x86_64 5.2.6-1.el5.art atomic
php-gd.x86_64 5.2.6-1.el5.art atomic
php-imap.x86_64 5.2.6-1.el5.art atomic
php-mbstring.x86_64 5.2.6-1.el5.art atomic
php-mysql.x86_64 5.2.6-1.el5.art atomic
php-pdo.x86_64 5.2.6-1.el5.art atomic
php-xml.x86_64 5.2.6-1.el5.art atomic
psa-kronolith.noarch 2.1.8-1.el5.art atomic
razor-agents.x86_64 2.84-1.el5.art atomic
spamassassin.x86_64 1:3.2.5-1.el5.art atomic

It's interesting to note that httpd and mod_ssl won't update with yum and gives this error:

Running rpm_check_debug
ERROR with rpm_check_debug vs depsolve:
Package psa-tomcat-configurator needs mod_jk, this is not available.
Complete!
 
Im afraid there is no way to condense my 15+ years of forensics investigation experience into a forum post :p Needless to say, you my advice is to look for anything "weird". From the information you've posted so far it sounds to me like someone is doing something nefarious. I suggest you try capturing it with a sniffer to see whats going on. Generally a UDP flood would require root privileges, its certainly possible if they've got a rootkit on the box to hide that kind of activity from both ps and mod_top.
 
Thanks Scott,

I know you have good experience, maybe an option would be for me to make use of your professional services to look into this.

Will installing ASL be helpful at all in a case like this?

What do you make of httpd and mod_ssl not wanting to update and will it be safe to do so using yum?
 
Greetings:

While rootkit hunter, chkrootkit, and ossec rootcheck are good root kit detection tools, they typically only find root kits. In our experience, most of the attacks in the past several years center around web-based injection attacks.

The code for such attacks can be in any directory the web sever can write. This can range from /tmp, /var/tmp, /dev/shm, /var/spool/samba, /var/spool/vbox, /var/spool/squid, and /var/spool/cron along with the web site home directories.

Sometimes Clam Anti-Virus clamscan can find such malware with the --infected and --recursive options.

Sometimes you can use the "ps -efl" (or ps -auwx if FreeBSD) to go through the process tree and look for suspicious processes.

Thank you.
 
Thanks Peter,

I did some previous checks with chkrootkit and rootkit hunter but only had a few warnings.
I now installed ossec-rootcheck and it gave this:

[INFO]: Starting rootcheck scan.

[OK]: No presence of public rootkits detected. Analyzed 270 files.

[OK]: No binaries with any trojan detected. Analyzed 79 files.

[INFO]: System Audit: Web exploits (uncommon file name inside htdocs) - Possible compromise. File: /var/www/vhosts/chroot/bin/id. Reference: http://www.ossec.net/wiki/index.php/WebAttacks_links .

[INFO]: System Audit: Web exploits (uncommon file name inside htdocs) - Possible compromise. File: /var/www/.ssh. Reference: http://www.ossec.net/wiki/index.php/WebAttacks_links .

[OK]: No problem detected on the /dev directory. Analyzed 227 files

[FAILED]: File '/usr/local/psa/admin/htdocs/powertoys/emailtemplates/send_password_subject.html' is:
- owned by root,
- has written permissions to anyone.

[FAILED]: File '/usr/local/psa/admin/htdocs/powertoys/emailtemplates/send_password_html.html' is:
- owned by root,
- has written permissions to anyone.

[FAILED]: File '/usr/local/psa/admin/htdocs/powertoys/emailtemplates/domain_will_expire_html.html' is:
- owned by root,
- has written permissions to anyone.

[FAILED]: File '/usr/local/psa/admin/htdocs/powertoys/emailtemplates/domain_will_expire_txt.html' is:
- owned by root,
- has written permissions to anyone.

[FAILED]: File '/usr/local/psa/admin/htdocs/powertoys/emailtemplates/domain_expire_txt.html' is:
- owned by root,
- has written permissions to anyone.

[FAILED]: File '/usr/local/psa/admin/htdocs/powertoys/emailtemplates/domain_expire_html.html' is:
- owned by root,
- has written permissions to anyone.

[FAILED]: File '/usr/local/psa/admin/htdocs/powertoys/emailtemplates/domain_expire_subject.html' is:
- owned by root,
- has written permissions to anyone.

[FAILED]: File '/usr/local/psa/admin/htdocs/powertoys/emailtemplates/send_password_txt.html' is:
- owned by root,
- has written permissions to anyone.

[FAILED]: File '/usr/local/psa/admin/htdocs/powertoys/licdata.php' is:
- owned by root,
- has written permissions to anyone.

[FAILED]: File '/usr/local/psa/admin/htdocs/powertoys/key.php' is:
- owned by root,
- has written permissions to anyone.

[FAILED]: File '/usr/local/psa/tmp/vhosts.tar.gz' is:
- owned by root,
- has written permissions to anyone.

[FAILED]: File '/usr/local/psa/tmp/default_skeleton.tgz' is:
- owned by root,
- has written permissions to anyone.

[FAILED]: File '/usr/local/psa/tmp/ftp.pamd' is:
- owned by root,
- has written permissions to anyone.

[FAILED]: File '/usr/local/psa/tmp/psa.key' is:
- owned by root,
- has written permissions to anyone.

[FAILED]: File '/usr/local/psa/tmp/run-root.tar' is:
- owned by root,
- has written permissions to anyone.

[FAILED]: File '/usr/local/psa/var/modules/watchdog/lib/rkhunter/db/mirrors.dat' is:
- owned by root,
- has written permissions to anyone.

[FAILED]: File '/sys/module/sbs/parameters/capacity_mode' is:
- owned by root,
- has written permissions to anyone.

[FAILED]: File '/sys/module/sbs/parameters/update_mode' is:
- owned by root,
- has written permissions to anyone.

[ERR]: Check the following files for more information:
rootcheck-rw-rw-rw-.txt (list of world writable files)
rootcheck-rwxrwxrwx.txt (list of world writtable/executable files)
rootcheck-suid-files.txt (list of suid files)

[OK]: No hidden process by Kernel-level rootkits.
/bin/ps is not trojaned. Analyzed 32768 processes.

[OK]: No kernel-level rootkit hiding any port.
Netstat is acting correctly. Analyzed 131072 ports.

[OK]: The following ports are open:
21 (tcp),22 (tcp),25 (tcp),53 (tcp),53 (udp),
80 (tcp),106 (tcp),110 (tcp),111 (tcp),111 (udp),
143 (tcp),443 (tcp),465 (tcp),953 (tcp),
993 (tcp),995 (tcp),1701 (tcp),3000 (tcp),
3306 (tcp),5353 (udp),5432 (tcp),8443 (tcp),
8880 (tcp),32859 (udp),32860 (udp),33301 (udp),
33483 (udp)

[OK]: No problem detected on ifconfig/ifs. Analyzed 5 interfaces.


- Scan completed in 145 seconds.
[INFO]: Ending rootcheck scan.
 
rkhunter output - only warnings:


Checking rkhunter data files...
Checking file mirrors.dat [ No update ]
Checking file programs_bad.dat [ No update ]
Checking file backdoorports.dat [ No update ]
Checking file suspscan.dat [ No update ]
Checking file i18n/cn [ No update ]
Checking file i18n/en [ No update ]
Checking file i18n/zh [ No update ]
Checking file i18n/zhutf [ No update ]

Checking system commands...
Performing 'strings' command checks
Checking 'strings' command [ OK ]

Performing 'shared libraries' checks
Checking for preloading variables [ None found ]
Checking for preload file [ Not found ]
Checking LD_LIBRARY_PATH variable [ Not found ]
Performing file properties checks
Checking for prerequisites [ Warning ]

/usr/bin/GET [ Warning ]
/usr/bin/groups [ Warning ]

/usr/bin/ldd [ Warning ]

/usr/bin/whatis [ Warning ]

/sbin/ifdown [ Warning ]
/sbin/ifup [ Warning ]

Performing trojan specific checks
Checking for enabled xinetd services [ Warning ]
Checking for Apache backdoor [ Not found ]

Performing group and account checks
Checking for passwd file [ Found ]
Checking for root equivalent (UID 0) accounts [ None found ]
Checking for passwordless accounts [ None found ]
Checking for passwd file changes [ Warning ]
Checking for group file changes [ Warning ]
Checking root account shell history files [ OK ]

Performing filesystem checks
Checking /dev for suspicious file types [ None found ]
Checking for hidden files and directories [ Warning ]

Checking application versions...

Checking version of GnuPG [ OK ]
Checking version of Apache [ OK ]
Checking version of Bind DNS [ OK ]
Checking version of OpenSSL [ OK ]
Checking version of PHP [ OK ]
Checking version of Procmail MTA [ OK ]
Checking version of ProFTPd [ OK ]
Checking version of OpenSSH [ OK ]
 
Greetings Ryan:

Please note I did not careful examine the output (our clients pay us to clean servers). What I would check is the various directories mentioned, run Clam Scan with the options provided, and check the process tree for suspicious files.

It most likely is not a root kit, though every area of caution / warning / error for ossec-rootcheck should be examined if the settings are exactly as needed.

Thank you.
 
Thanks for your advice Peter,

We just needed some directions and pointers. We'll get someone to check it out.

Clamscan comes up clean
 
Hi Ryan:

Understood. Chances are high it is an application in an end user directory or other directory for which the web server can write rather than a root kit.

Thank you.
 
Yo may wish to run netstat -anp which will show a list of processes and open ports... you can then check in /proc/"processid" for further information about the running processes of interest.
 
You must log in or register to reply here.
Back
Top