1. Please take a little time for this simple survey! Thank you for participating!
    Dismiss Notice
  2. Dear Pleskians, please read this carefully! New attachments and other rules Thank you!
    Dismiss Notice
  3. Dear Pleskians, I really hope that you will share your opinion in this Special topic for chatter about Plesk in the Clouds. Thank you!
    Dismiss Notice

Possibly Hacked - High Traffic and Httpd

Discussion in 'Plesk for Linux - 8.x and Older' started by ryanz, Jul 21, 2008.

  1. ryanz

    ryanz Basic Pleskian

    24
    73%
    Joined:
    Nov 23, 2002
    Messages:
    91
    Likes Received:
    0
    Hi,

    I'm not sure what the cause is or if it's coincidence but our server stared behaving strangely just a few days after the upgrade to 8.4.0 on Cent 0S 5-64.

    The outgoing traffic spikes to 50-80Mbs for an hour or so then goes back to normal for a while then just spikes back up again.

    During this time TOP shows 2 httpd processes running at 99% CPU and iftop shows high outgoing traffic to a number of domains and ips. Ntop shows outgoing traffic to be UDP but we cannot find the source.

    We have APF, BFD, Mod_Security, chkrootkit, rkhunter running with a secured /tmp and can't find anything obvious.

    Has anyone had this experience and what are the steps to follow to try and solve this?
     
  2. atomicturtle

    atomicturtle Golden Pleskian

    29
     
    Joined:
    Nov 20, 2002
    Messages:
    2,110
    Likes Received:
    7
    Location:
    Washington, DC
    That is pretty suspicious, you could try grabbing a sniffer dump the next time it happens. Another good tool is www.mod-top.org, which will let you look into what apache is doing.

    A forensic first step would be to run rpm -Va on the system, and look into all the changes against the base rpm settings. There will be a lot, its not the kind of thing where there are any shortcuts I'm afraid. Second, any standard rootkit will more or less be able to hide from that anyway.
     
  3. filippo.toso

    filippo.toso Guest

    0
     
    This is a kind of "leap in the dark" but if you aren't using the DNS service you can try to disable it. I'm experiencing the same problem since Sunday night and after a lot of tests I have found that disabling the DNS service in Plesk, the traffic just dropped to the normal amount.
     
  4. atomicturtle

    atomicturtle Golden Pleskian

    29
     
    Joined:
    Nov 20, 2002
    Messages:
    2,110
    Likes Received:
    7
    Location:
    Washington, DC
    All that means is you're disabling DNS lookups for whatever is going on. It would certainly break anything nefarious going on that was DNS based, at least until the badguys notice and change it back :p
     
  5. ryanz

    ryanz Basic Pleskian

    24
    73%
    Joined:
    Nov 23, 2002
    Messages:
    91
    Likes Received:
    0
    Thanks for the suggestions so far.

    I did a test and closed all UDP ports in APF, since then there seems to be no problem with the server running at normal traffic for about 10 hours.

    Any suggestions?

    Scott, what should I be looking for with rpm -Va? Here are some details:
    .M...... /usr/bin/curl
    .......T c /etc/libuser.conf
    .......T c /etc/mail/sendmail.cf
    S.5....T c /var/log/mail/statistics
    S.5....T c /etc/ssh/sshd_config
    .......T d /usr/share/man/man1/afs5log.1.gz
    .......T d /usr/share/man/man5/pam_krb5.5.gz
    .......T d /usr/share/man/man8/pam_krb5.8.gz
    .......T d /usr/share/man/man8/pam_krb5_storetmp.8.gz
    .M...... /usr/bin/wget
    ..5....T c /etc/pki/nssdb/secmod.db
    .......T /usr/share/X11/XErrorDB
    S.5....T c /etc/php.ini
    .......T /usr/include/pcap-bpf.h
    .......T /usr/include/pcap-namedb.h
    .......T /usr/include/pcap.h
    prelink: /usr/bin/pk11install: at least one of file's dependencies has changed since prelinking
    S.?..... /usr/bin/pk11install
     
  6. ryanz

    ryanz Basic Pleskian

    24
    73%
    Joined:
    Nov 23, 2002
    Messages:
    91
    Likes Received:
    0
    I installed mod_top and ran it for a while to see the processes, then I opened the UDP ports in APF and after a few minutes there was high httpd/apache load with very high traffic.

    TOP showed two httpd processes running at 99% cpu each but mod_top showed only one running at 28% cpu.

    If I restart or stop httpd while the load and traffic is high everything returns to normal and then simply goes up again after a few minutes.

    We get problems when we open these ports:
    IG_UDP_CPORTS="37,53,161,873"
    EG_UDP_CPORTS="53,873"
     
  7. ryanz

    ryanz Basic Pleskian

    24
    73%
    Joined:
    Nov 23, 2002
    Messages:
    91
    Likes Received:
    0
    By the way, I installed Plesk using the Autoinstaller and also did the upgrade to 8.4.0 using the auto updater in the CP.

    According to yum check-update these are the only packages that are not updated and I'm worried they might break something or cause problems with future auto updates.

    httpd.x86_64 2.2.3-11.el5_1.centos. base
    kbd.x86_64 1.12-20.el5 base
    kernel.x86_64 2.6.18-92.1.6.el5 updates
    kernel-devel.x86_64 2.6.18-92.1.6.el5 updates
    kernel-headers.x86_64 2.6.18-92.1.6.el5 updates
    libhugetlbfs.x86_64 1.2-5.el5 base
    mod_ssl.x86_64 1:2.2.3-11.el5_1.cento base
    mysql.i386 5.0.45-7.el5 base
    mysql.x86_64 5.0.58-1.el5.art atomic
    mysql-server.x86_64 5.0.58-1.el5.art atomic
    php.x86_64 5.2.6-1.el5.art atomic
    php-cli.x86_64 5.2.6-1.el5.art atomic
    php-common.x86_64 5.2.6-1.el5.art atomic
    php-gd.x86_64 5.2.6-1.el5.art atomic
    php-imap.x86_64 5.2.6-1.el5.art atomic
    php-mbstring.x86_64 5.2.6-1.el5.art atomic
    php-mysql.x86_64 5.2.6-1.el5.art atomic
    php-pdo.x86_64 5.2.6-1.el5.art atomic
    php-xml.x86_64 5.2.6-1.el5.art atomic
    psa-kronolith.noarch 2.1.8-1.el5.art atomic
    razor-agents.x86_64 2.84-1.el5.art atomic
    spamassassin.x86_64 1:3.2.5-1.el5.art atomic

    It's interesting to note that httpd and mod_ssl won't update with yum and gives this error:

    Running rpm_check_debug
    ERROR with rpm_check_debug vs depsolve:
    Package psa-tomcat-configurator needs mod_jk, this is not available.
    Complete!
     
  8. atomicturtle

    atomicturtle Golden Pleskian

    29
     
    Joined:
    Nov 20, 2002
    Messages:
    2,110
    Likes Received:
    7
    Location:
    Washington, DC
    Im afraid there is no way to condense my 15+ years of forensics investigation experience into a forum post :p Needless to say, you my advice is to look for anything "weird". From the information you've posted so far it sounds to me like someone is doing something nefarious. I suggest you try capturing it with a sniffer to see whats going on. Generally a UDP flood would require root privileges, its certainly possible if they've got a rootkit on the box to hide that kind of activity from both ps and mod_top.
     
  9. ryanz

    ryanz Basic Pleskian

    24
    73%
    Joined:
    Nov 23, 2002
    Messages:
    91
    Likes Received:
    0
    Thanks Scott,

    I know you have good experience, maybe an option would be for me to make use of your professional services to look into this.

    Will installing ASL be helpful at all in a case like this?

    What do you make of httpd and mod_ssl not wanting to update and will it be safe to do so using yum?
     
  10. dynamicnet

    dynamicnet Basic Pleskian

    23
    90%
    Joined:
    Sep 19, 2007
    Messages:
    65
    Likes Received:
    0
    Greetings:

    While rootkit hunter, chkrootkit, and ossec rootcheck are good root kit detection tools, they typically only find root kits. In our experience, most of the attacks in the past several years center around web-based injection attacks.

    The code for such attacks can be in any directory the web sever can write. This can range from /tmp, /var/tmp, /dev/shm, /var/spool/samba, /var/spool/vbox, /var/spool/squid, and /var/spool/cron along with the web site home directories.

    Sometimes Clam Anti-Virus clamscan can find such malware with the --infected and --recursive options.

    Sometimes you can use the "ps -efl" (or ps -auwx if FreeBSD) to go through the process tree and look for suspicious processes.

    Thank you.
     
  11. ryanz

    ryanz Basic Pleskian

    24
    73%
    Joined:
    Nov 23, 2002
    Messages:
    91
    Likes Received:
    0
    Thanks Peter,

    I did some previous checks with chkrootkit and rootkit hunter but only had a few warnings.
    I now installed ossec-rootcheck and it gave this:

    [INFO]: Starting rootcheck scan.

    [OK]: No presence of public rootkits detected. Analyzed 270 files.

    [OK]: No binaries with any trojan detected. Analyzed 79 files.

    [INFO]: System Audit: Web exploits (uncommon file name inside htdocs) - Possible compromise. File: /var/www/vhosts/chroot/bin/id. Reference: http://www.ossec.net/wiki/index.php/WebAttacks_links .

    [INFO]: System Audit: Web exploits (uncommon file name inside htdocs) - Possible compromise. File: /var/www/.ssh. Reference: http://www.ossec.net/wiki/index.php/WebAttacks_links .

    [OK]: No problem detected on the /dev directory. Analyzed 227 files

    [FAILED]: File '/usr/local/psa/admin/htdocs/powertoys/emailtemplates/send_password_subject.html' is:
    - owned by root,
    - has written permissions to anyone.

    [FAILED]: File '/usr/local/psa/admin/htdocs/powertoys/emailtemplates/send_password_html.html' is:
    - owned by root,
    - has written permissions to anyone.

    [FAILED]: File '/usr/local/psa/admin/htdocs/powertoys/emailtemplates/domain_will_expire_html.html' is:
    - owned by root,
    - has written permissions to anyone.

    [FAILED]: File '/usr/local/psa/admin/htdocs/powertoys/emailtemplates/domain_will_expire_txt.html' is:
    - owned by root,
    - has written permissions to anyone.

    [FAILED]: File '/usr/local/psa/admin/htdocs/powertoys/emailtemplates/domain_expire_txt.html' is:
    - owned by root,
    - has written permissions to anyone.

    [FAILED]: File '/usr/local/psa/admin/htdocs/powertoys/emailtemplates/domain_expire_html.html' is:
    - owned by root,
    - has written permissions to anyone.

    [FAILED]: File '/usr/local/psa/admin/htdocs/powertoys/emailtemplates/domain_expire_subject.html' is:
    - owned by root,
    - has written permissions to anyone.

    [FAILED]: File '/usr/local/psa/admin/htdocs/powertoys/emailtemplates/send_password_txt.html' is:
    - owned by root,
    - has written permissions to anyone.

    [FAILED]: File '/usr/local/psa/admin/htdocs/powertoys/licdata.php' is:
    - owned by root,
    - has written permissions to anyone.

    [FAILED]: File '/usr/local/psa/admin/htdocs/powertoys/key.php' is:
    - owned by root,
    - has written permissions to anyone.

    [FAILED]: File '/usr/local/psa/tmp/vhosts.tar.gz' is:
    - owned by root,
    - has written permissions to anyone.

    [FAILED]: File '/usr/local/psa/tmp/default_skeleton.tgz' is:
    - owned by root,
    - has written permissions to anyone.

    [FAILED]: File '/usr/local/psa/tmp/ftp.pamd' is:
    - owned by root,
    - has written permissions to anyone.

    [FAILED]: File '/usr/local/psa/tmp/psa.key' is:
    - owned by root,
    - has written permissions to anyone.

    [FAILED]: File '/usr/local/psa/tmp/run-root.tar' is:
    - owned by root,
    - has written permissions to anyone.

    [FAILED]: File '/usr/local/psa/var/modules/watchdog/lib/rkhunter/db/mirrors.dat' is:
    - owned by root,
    - has written permissions to anyone.

    [FAILED]: File '/sys/module/sbs/parameters/capacity_mode' is:
    - owned by root,
    - has written permissions to anyone.

    [FAILED]: File '/sys/module/sbs/parameters/update_mode' is:
    - owned by root,
    - has written permissions to anyone.

    [ERR]: Check the following files for more information:
    rootcheck-rw-rw-rw-.txt (list of world writable files)
    rootcheck-rwxrwxrwx.txt (list of world writtable/executable files)
    rootcheck-suid-files.txt (list of suid files)

    [OK]: No hidden process by Kernel-level rootkits.
    /bin/ps is not trojaned. Analyzed 32768 processes.

    [OK]: No kernel-level rootkit hiding any port.
    Netstat is acting correctly. Analyzed 131072 ports.

    [OK]: The following ports are open:
    21 (tcp),22 (tcp),25 (tcp),53 (tcp),53 (udp),
    80 (tcp),106 (tcp),110 (tcp),111 (tcp),111 (udp),
    143 (tcp),443 (tcp),465 (tcp),953 (tcp),
    993 (tcp),995 (tcp),1701 (tcp),3000 (tcp),
    3306 (tcp),5353 (udp),5432 (tcp),8443 (tcp),
    8880 (tcp),32859 (udp),32860 (udp),33301 (udp),
    33483 (udp)

    [OK]: No problem detected on ifconfig/ifs. Analyzed 5 interfaces.


    - Scan completed in 145 seconds.
    [INFO]: Ending rootcheck scan.
     
  12. ryanz

    ryanz Basic Pleskian

    24
    73%
    Joined:
    Nov 23, 2002
    Messages:
    91
    Likes Received:
    0
    rkhunter output - only warnings:


    Checking rkhunter data files...
    Checking file mirrors.dat [ No update ]
    Checking file programs_bad.dat [ No update ]
    Checking file backdoorports.dat [ No update ]
    Checking file suspscan.dat [ No update ]
    Checking file i18n/cn [ No update ]
    Checking file i18n/en [ No update ]
    Checking file i18n/zh [ No update ]
    Checking file i18n/zhutf [ No update ]

    Checking system commands...
    Performing 'strings' command checks
    Checking 'strings' command [ OK ]

    Performing 'shared libraries' checks
    Checking for preloading variables [ None found ]
    Checking for preload file [ Not found ]
    Checking LD_LIBRARY_PATH variable [ Not found ]
    Performing file properties checks
    Checking for prerequisites [ Warning ]

    /usr/bin/GET [ Warning ]
    /usr/bin/groups [ Warning ]

    /usr/bin/ldd [ Warning ]

    /usr/bin/whatis [ Warning ]

    /sbin/ifdown [ Warning ]
    /sbin/ifup [ Warning ]

    Performing trojan specific checks
    Checking for enabled xinetd services [ Warning ]
    Checking for Apache backdoor [ Not found ]

    Performing group and account checks
    Checking for passwd file [ Found ]
    Checking for root equivalent (UID 0) accounts [ None found ]
    Checking for passwordless accounts [ None found ]
    Checking for passwd file changes [ Warning ]
    Checking for group file changes [ Warning ]
    Checking root account shell history files [ OK ]

    Performing filesystem checks
    Checking /dev for suspicious file types [ None found ]
    Checking for hidden files and directories [ Warning ]

    Checking application versions...

    Checking version of GnuPG [ OK ]
    Checking version of Apache [ OK ]
    Checking version of Bind DNS [ OK ]
    Checking version of OpenSSL [ OK ]
    Checking version of PHP [ OK ]
    Checking version of Procmail MTA [ OK ]
    Checking version of ProFTPd [ OK ]
    Checking version of OpenSSH [ OK ]
     
  13. dynamicnet

    dynamicnet Basic Pleskian

    23
    90%
    Joined:
    Sep 19, 2007
    Messages:
    65
    Likes Received:
    0
    Greetings Ryan:

    Please note I did not careful examine the output (our clients pay us to clean servers). What I would check is the various directories mentioned, run Clam Scan with the options provided, and check the process tree for suspicious files.

    It most likely is not a root kit, though every area of caution / warning / error for ossec-rootcheck should be examined if the settings are exactly as needed.

    Thank you.
     
  14. ryanz

    ryanz Basic Pleskian

    24
    73%
    Joined:
    Nov 23, 2002
    Messages:
    91
    Likes Received:
    0
    Thanks for your advice Peter,

    We just needed some directions and pointers. We'll get someone to check it out.

    Clamscan comes up clean
     
  15. dynamicnet

    dynamicnet Basic Pleskian

    23
    90%
    Joined:
    Sep 19, 2007
    Messages:
    65
    Likes Received:
    0
    Hi Ryan:

    Understood. Chances are high it is an application in an end user directory or other directory for which the web server can write rather than a root kit.

    Thank you.
     
  16. sbillis

    sbillis Basic Pleskian

    24
     
    Joined:
    Mar 2, 2007
    Messages:
    40
    Likes Received:
    1
    Yo may wish to run netstat -anp which will show a list of processes and open ports... you can then check in /proc/"processid" for further information about the running processes of interest.
     
Loading...