Resolved Nginx error log shows foreign domains pointed to our server.

[afuego]

Server operating system version

Ubuntu 20.04.5

Plesk version and microupdate number

18.0.49

Last night some our websites went down with 503 errors due to MaxRequestWorkers server limit reached, reviewing the Nginx error log, I noticed about 8 different domains listed as "host" that point to our public IP address.

Why would someone point these domains to our server without our knowledge? How do we lock it down? Should we block the client IP address or the domains?

Nginx error
root@plesk:~# cat /var/log/nginx/error.log |grep "Connection timed out" |tail -1
2023/01/18 05:23:39 [error] 557940#0: *12873929 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 3.89.164.170, server: , request: "GET / HTTP/1.1", upstream: "http://our-internal-ip:7080/", host: "foreigndomain.com"

Apache error
root@plesk:~# cat /var/log/apache2/error.log |grep ServerLimit | tail -1
[Wed Jan 18 05:24:05.421117 2023] [mpm_event:error] [pid 1601908:tid 140030268120128] AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit.

Thank you for your advice!

 

[Peter Debik]

It could be happening by accident because someone on the Internet misconfigured the IP address. But sometimes this could also be a way to attack a server. You can block requests for the foreign domain that are directed to your server by adding this to iptables:

# iptables -I INPUT 1 -p tcp -m multiport --dports 80,443,7080,7081,8443,8447 -m string --algo bm --string "DOMAIN.TLD" -j REJECT --reject-with tcp-reset

# iptables -I FORWARD 1 -p tcp -m multiport --dports 80,443,7080,7081,8443,8447 -m string --algo bm --string "DOMAIN.TLD" -j REJECT --reject-with tcp-reset

# iptables -I OUTPUT 1 -p tcp -m multiport --dports 80,443,7080,7081,8443,8447 -m string --algo bm --string "DOMAIN.TLD" -j REJECT --reject-with tcp-reset

Replace "DOMAIN.TLD" with the foreign domain for that you wish to block all traffic.

 

[afuego]

Thank you so much @Peter Debik

 

[mow]

It could be happening by accident because someone on the Internet misconfigured the IP address. But sometimes this could also be a way to attack a server. You can block requests for the foreign domain that are directed to your server by adding this to iptables:

# iptables -I INPUT 1 -p tcp -m multiport --dports 80,443,7080,7081,8443,8447 -m string --algo bm --string "DOMAIN.TLD" -j REJECT --reject-with tcp-reset

# iptables -I FORWARD 1 -p tcp -m multiport --dports 80,443,7080,7081,8443,8447 -m string --algo bm --string "DOMAIN.TLD" -j REJECT --reject-with tcp-reset

# iptables -I OUTPUT 1 -p tcp -m multiport --dports 80,443,7080,7081,8443,8447 -m string --algo bm --string "DOMAIN.TLD" -j REJECT --reject-with tcp-reset

Replace "DOMAIN.TLD" with the foreign domain for that you wish to block all traffic.

Why all three? Shouldn't INPUT be enough?
This could take considerable processing power because all packets are screened for the (sub)string, although on the SSL ports you can only get a hit for connections in the SNI phase of the negotiation.
It would probably be more efficient to add the domain(s) as a trigger to fail2ban.

 

[Peter Debik]

Why all three? Shouldn't INPUT be enough?

Yes, it should. But I have seen cases here where the "attack" was actually not started from the outside but from a website on the server itself, asking another server to request more data. For that reason it is best to block all options.

This could take considerable processing power because all packets are screened for the (sub)string, although on the SSL ports you can only get a hit for connections in the SNI phase of the negotiation.

Yes, these rules are using a lot of cpu power alone for the "bm" option. But it is still minimal as there are not many of them.

It would probably be more efficient to add the domain(s) as a trigger to fail2ban.

Please provide the exact steps that are needed for that alternative solution. I certainly agree that doing this through Fail2Ban can also be a good approach. However, my approach stops the issue for sure right now while fail2ban needs configuration and testing time (for a new jail for example). I prefer the "stop it now" solution in such cases, but another solution can also work.

 

[mow]

Please provide the exact steps that are needed for that alternative solution. I certainly agree that doing this through Fail2Ban can also be a good approach. However, my approach stops the issue for sure right now while fail2ban needs configuration and testing time (for a new jail for example). I prefer the "stop it now" solution in such cases, but another solution can also work.

I have not used fail2ban myself yet, but from what I gathered it scans the logs for keywords (failed logins, attempts to access certain URLs from exploits ...) and bans the IP associated with it. So if that domainname was added as a bad word it would block any IP attempting to access that domain. That might be a bit harsh, but efficient as an IP filter costs far less CPU than a substring search.

However both approaches won't work for domains that also normally occur in traffic, so one should be careful.