Issue Unbanning some IP addresses causes the server to be unresponsive

[TurabG]

Server operating system version

Almalinux 8

Plesk version and microupdate number

Plesk 18.0.63 #4

I have a firewall in front of the Plesk web server. (Endian Community) My web server is not very active, it has only 10-15 sites whose traffic shouldn't be more than 40-50GB monthly but I see 500-700GB traffic due to "bad bots". So from time to time, I go to Fail2Ban interface to look for IP patterns to block them on firewall level. For example I block cloud services (like Amazon, Google, Microsoft) (not the search engines) and suspicious IP addresses (as full ranges) that belong to suspicious organizations and that I am sure are not individual users.

Then I remove them from Fail2Ban block list as they are now blocked on an upper level which can't reach here. But something strange happens lately. When I select a few IP addresses and click unban button, the whole server gets inaccessible for a minute or two. None of the websites response for a couple of minutes. Then when the server becomes accesible back, I try removing those IPs again, and most the times it works the second time. So I check a few more IP addresses and click unban again, the server becomes inaccessible again..

What could be causing this, how can I even diagnose it?

If I have an active SSH session, I see that it doesn't get cut off during this time. But if I don't have an active SSH session before I clicked unban button, I cannot make a SSH connection during this time. Isn't that strange?

 

[La Linea]

How many firewall rules do you have and how do you block those cloud servies and suspicious IP addresses on the server?

Do you use plesk firewall or have you configured it manually? How many cores and RAM does the server have and what are the load and ram usage averages?

And no, "bad bots" don't cause that much additional traffic.

 

[TurabG]

And no, "bad bots" don't cause that much additional traffic.

Well they do; as I can see them in Awstats and I don't see them when I block the known bad-bots IP block. After I had done some blocking, the traffic dropped back to approx. 30-40GBs.

By the way, as I said before, I have blocked some large IP pools of some cloud services like Google's, Amazon's, Digital Oceans, Vultr's etc.. not the legit bots or search engines, I mean the cloud services that they hire to they customers, who in turn use those IPs at their own disposal. (They all officially publish their customer pools; that's where I get the list.) So for example why would a Digital Ocean customer's cloud box should send automated requests to my server? That's the reason behind my blocking them.

But turns out unblocking all the IPs from the firewall seems to have resolved the aforementioned issue on this topic. And I don't understand how. My data center says I could have blocked some DNS servers that some services needed; but this explanation is not sufficient and certain. I am still looking for the root cause.

 

[La Linea]

Well they do; as I can see them in Awstats and I don't see them when I block the known bad-bots IP block. After I had done some blocking, the traffic dropped back to approx. 30-40GBs.

When your server is under some kind of attack from within these address ranges, you will of course see a lot of (incoming) traffic, but that's not traffic from the systems targeted within the Fail2Ban badbots filter....

By the way, as I said before, I have blocked some large IP pools of some cloud services like Google's, Amazon's, Digital Oceans, Vultr's etc.. not the legit bots or search engines, I mean the cloud services that they hire to they customers, who in turn use those IPs at their own disposal. (They all officially publish their customer pools; that's where I get the list.) So for example why would a Digital Ocean customer's cloud box should send automated requests to my server? That's the reason behind my blocking them..

Instead of explaining at length WHY you block these addresses, which is completely irrelevant to the problem described, you would have been better off answering the questions about HOW you block them.

But turns out unblocking all the IPs from the firewall seems to have resolved the aforementioned issue on this topic. And I don't understand how.

That's exactly why I was asking you how many firewall rules you have, how you block those cloud servies and suspicious IP addresses and what the load and ram usage averages are when you are experiencing such an unresponiveness ...

My data center says I could have blocked some DNS servers that some services needed; but this explanation is not sufficient and certain. I am still looking for the root cause.

This sounds pretty much like a typical "We will waste your time, when you try to waste our time with your problems"-response in my opinion. After all, no service on your system needs any other DNS server than the one(s) you have configured in resolv.conf and unless any of the servers configured there, as well as your server itself, are within any of the ranges you have blocked, which I expect you have made sure of in the first place, you can simply ignore this response from your data center.

 

[TurabG]

Thank you for taking time to concern.

When your server is under some kind of attack from within these address ranges, you will of course see a lot of (incoming) traffic, but that's not traffic from the systems targeted within the Fail2Ban badbots filter....

It's not an attack, it's a regular bot visit that I can see on the stats. To be exact, one named "feed" in the bot list caused most of this traffic. For 3-4 months, the traffic was between 500-700GB regularly. And they didn't cause any black-outs on the server; they just ate up the traffic. After I blocked large IP pools (which I collected from Fail2ban), the traffic dropped back to 20-30GBs.

Instead of explaining at length WHY you block these addresses, which is completely irrelevant to the problem described, you would have been better off answering the questions about HOW you block them.

I already told that I was using Endian Community Firewall, in front of the web server. So the requests coming from those IPs were dropped at the firewall level, before it ever reached to Plesk box.

This sounds pretty much like a typical "We will waste your time, when you try to waste our time with your problems"-response in my opinion. After all, no service on your system needs any other DNS server than the one(s) you have configured in resolv.conf and unless any of the servers configured there, as well as your server itself, are within any of the ranges you have blocked, which I expect you have made sure of in the first place

Yes, quite I thought. Nevertheless, it still could be caused by something I did but I am still not getting how. I also have VPN set on Endian and I have Adguard Home DNS that I setup which uses Cloudflare's famous 1.1.1.1 as an upstream DNS server. Both the DNS and my network is set to use this Adguard instance. Maybe I set something wrong in firewall's DNS but I can't confirm this is the case because I have changed absolutely none of the settings on the firewall; neither for DNS nor for any other things. I just blocked IPs.

That's exactly why I was asking you how many firewall rules you have, how you block those cloud servies and suspicious IP addresses and what the load and ram usage averages are when you are experiencing such an unresponiveness ...

I set a couple of rules to drop packets coming from some IP addresses which include some very large pools (like some /12, /18 classes). And there was still no problem. Maybe the very last IP pools I added caused the problem but as I said I don't see how. The amount of the rules are not much on the firewall. The amount of IPs are about 300. (But as I said they aren't of /32 class.)