Out of Control SSL log file - grows by GB a day

[JeffreyZ]

I suddenly have a client's access_SSL_log.processed file growing at a ridiculous rate. Very small website. The log file had grown to 6BG before I realized the problem. I removed it and two days later there was a 3GB log file that replaced it. Nothing strange that I can see regarding the traffic to the website, which is quite small. Only 80 visitors.

I looked at the end of the log file but I don't understand how to read it.

I've shut off logging for now.

 

[Faris Raouf]

ON the face of it, this looks like an attempt to attack/compromise the site . All the IPs look different in your screen shot, so firewalling them may be difficult

 

[LeonardChallis]

I had a fairly similar thing with access so I added some rules to iptables (firewall) to block anything apart from myself to SSH:

iptables -A INPUT -m state --state NEW,ESTABLISHED,RELATED --source x.x.x.x -p tcp --dport 22 -j ACCEPT
iptables -A INPUT -m state --state NEW,ESTABLISHED,RELATED -p tcp --dport 22 -j DROP

Here, x.x.x.x is your IP. You can add multiple by repeating the first line. Make sure you add them all before add the second rule to DROP every other SSH connection.

Note: If you make a mistake, then in virtuozzo control panel you can reset the firewall rules to default there

 

[atomicturtle]

Thats a heartbleed vulnerability attack. Its attempting to enumerate the sites private SSL key.

 

[JeffreyZ]

Thanks Faris, Leonard and atomicturtle! I'll try limiting IP access. I noticed that shutting off logging for that domain just moved the problem to proxy_access_ssl.log.

 

[JeffreyZ]

cannot restrict IPs

I tried the iptable restriction but that did not work. I then tried to restrict an ip using htaccess and that also did not work. I'm not sure if htaccess "deny" would even work in a log directory. The traffic seems to be on port 443.

One other thing that is odd to me is that this all started soon after I installed a new SSL certificate for Plesk logins.

And lastly, the traffic is from all over the world. No specific countries.

Seems it's resolved: I disabled the website for 5 minutes and then re-enabled it. Log files no longer growing. I also removed SSL support.

Prior to that I had submitted a support request to Parallels who replied:
=====
1. You can upgrade the Plesk Panel to 12.0.18 which contains failed to banned module. Failed to banned module, blocked the IP address automatically on the basis of rules, pattern of content and requests.
2. Change IP address of website to any dedicated IP.
3. Move or migrate this website to another server.
4. Contact website developer and check for SQL injections or malicious scripts into website coding.
5. You can switch off the logging for domain.
=====

 

[JeffreyZ]

The problem is not resolved. The next domain alphabetically now has the same problem of the rapidly growing log file. The original domain -- with SSL disabled -- no longer has the problem.

 

[Faris Raouf]

The iptables code that Leonard was suggesting would only work for SSH (port 22) not 443 (https). But anyway, as you say, the IPs are from all over the place and you can't restrict 443 otherwise nobody would be able to connect to secure sites.

Does this second site need https? If not, just switch it off. Do the same for the next site this happens to.

Basically, if I'm guessing correctly, you are being targeted by a botnet which is attempting to use the Heartbleed bug to obtain what I would presume to be as much data as possible -- obtaining a client's SSL key is not an issue unless they are a bank or something and the attacker intends to spam their users later. They are after login credentials, ideally Plesk ones.

The attacker would seem to have a list of some or all the domains on your Plesk box. They are probably trying them one by one.

While ASL (see www.atomicorp.com) would help you, it may not resolve the real issue, which is the massive logfiles.

How about changing logrotate settings so that they rotate by size (say 100Mb)? That would at least reduce the size of the logfiles to whatever size they may grow in 24 hours. I dare say it is possible to put together a script that runs every hour (either via logrotate or just using bash script) to look for ssl access logs that are larger than XMb and either deleting or rotating them.

The problem you are facing is going to hit a lot of us eventually, so finding the neatest solution we can as soon as we can would be useful.

 

[JeffreyZ]

fail2ban regex

Thanks Faris for your suggestions. I had the suggestion from Parallels to change my IP address. I had a suggestion from Media Temple to add a rule in fail2ban using regex. Unfortunately I find regex very hard to understand. It seems it could solve the problem, however.

So I've added this to fail2ban jail.local file but I don't know if the "action" line or the "failregex" lines are correct.

[ssh-ddos]
enabled = true
filter = sshd-ddos
action = iptables[name=SSH, port=ssh, protocol=tcp]
logpath = /var/www/vhosts/the-domain.com/logs/proxy_access_ssl_log
failregex = <HOST> - - .*x00*
maxretry = 3

All the log file entries start with x00 and any IP that is writing to this file will be a hack attempt, so I am trying simply to ban every IP that adds x00 to the log file.

The iptables code that Leonard was suggesting would only work for SSH (port 22) not 443 (https). But anyway, as you say, the IPs are from all over the place and you can't restrict 443 otherwise nobody would be able to connect to secure sites.

Does this second site need https? If not, just switch it off. Do the same for the next site this happens to.

Basically, if I'm guessing correctly, you are being targeted by a botnet which is attempting to use the Heartbleed bug to obtain what I would presume to be as much data as possible -- obtaining a client's SSL key is not an issue unless they are a bank or something and the attacker intends to spam their users later. They are after login credentials, ideally Plesk ones.

The attacker would seem to have a list of some or all the domains on your Plesk box. They are probably trying them one by one.

While ASL (see www.atomicorp.com) would help you, it may not resolve the real issue, which is the massive logfiles.

How about changing logrotate settings so that they rotate by size (say 100Mb)? That would at least reduce the size of the logfiles to whatever size they may grow in 24 hours. I dare say it is possible to put together a script that runs every hour (either via logrotate or just using bash script) to look for ssl access logs that are larger than XMb and either deleting or rotating them.

The problem you are facing is going to hit a lot of us eventually, so finding the neatest solution we can as soon as we can would be useful.