Issue All websites and plesk spontaneously unreachable

[damaverick]

Server operating system version

Almalinux 8.8

Plesk version and microupdate number

Latest Obsidion (I can't login in plesk right now to check)

Since yesterday evening all my websites and my plesk server are spontaneously unreachable by webbrowser.
Without even touching the server at all.

A few week ago I had exactly the same problem. Then I could fix it with restoring a backup from a day earlier despite the fact there was nothing done by me from the moment of the backup and the moment of the unreachable plesk environment.
This time restoring a backup didn't help at all.

I tried already several found solutions (fail2ban whitelisting, firewall whitelisting, setting nginx waiting time etc etc)
But as the websites are unreachable from everywhere, to me it looks like no fail2ban of 1 specific IP issue
But I don't know what the problem can be.
My server is reachable by ping and the ports for http, https and 8443 are open ports if I scan the server, but the websites and plesk portal isn't reachable.
The error in the browser I got was "the connection waitingtime has expired" and "504 gateway timeout from nginx"

I use Almalinux with Plesk on my VPS
And I don't know anymore how I could fix this and I hope someone could help me with this issue.

 

[damaverick]

Double post.... my apologies

 

[Peter Debik]

Are your webservers running?
# systemctl status nginx
# systemctl status httpd

 

[damaverick]

Are your webservers running?
# systemctl status nginx
# systemctl status httpd

Yes both are running

 

[Peter Debik]

A 504 gateway timeout points to an unresponsive Apache webserver or an unresponsive script that runs in the website. Have you checked the website's log files for errors?

 

[Maarten]

Can you check the /var/log/messages file for errors like this:

"named[xxxx]: no longer listening on xxx.xxx.xxx.xxx"
"A process of this unit has been killed by the OOM killer"

Are you using an IONOS VPS? Because I have seen some similar problems on this forum with VPSes from IONOS.

 

[damaverick]

A 504 gateway timeout points to an unresponsive Apache webserver or an unresponsive script that runs in the website. Have you checked the website's log files for errors?

I read a lot about that online indeed.
My issue doesn't look like it has to do with an unresponsive script.

I have had someone looking at it with me.
His conclusion is, it's a fail2ban/IP tables issue
it looks like fail2ban is filling the iptables wrongfully, which causes ports to be closed.

Is there anything changed lately in fail2ban of iptables? because this issue first appeared the end of august, and now 2 weeks later it spontaneously appeared again.

quick fix we did is manually change iptables from drop to accept.
This seems to work, but every reboot it's wrong again.

 

[damaverick]

Can you check the /var/log/messages file for errors like this:

"named[xxxx]: no longer listening on xxx.xxx.xxx.xxx"
"A process of this unit has been killed by the OOM killer"

Are you using an IONOS VPS? Because I have seen some similar problems on this forum with VPSes from IONOS.

no such errors were in the messages logs

I'm using transip VPS.
In my post above I describe our latest findings.

 

[Maarten]

There have been some updates in Plesk Obsidian 18.0.55 regarding Fail2ban:

Change Log for Plesk Obsidian

Find out about changes, additions and updates to Plesk Obsidian on an iteration to iteration basis.

docs.plesk.com

Please check if you have the plesk-modsecurity jail activated in Fail2ban. That can cause these kinds of issues. If it's active, disable it and see if it solves the problem.

Also, check if your VPS's IP addresses (IPv4/IPv6) are whitelisted in Fail2ban.

Btw: I know Transip. You don't have to expect a solution from them, even though they have to as a Plesk partner.

 

[damaverick]

There have been some updates in Plesk Obsidian 18.0.55 regarding Fail2ban:

Change Log for Plesk Obsidian

Find out about changes, additions and updates to Plesk Obsidian on an iteration to iteration basis.

docs.plesk.com

Please check if you have the plesk-modsecurity jail activated in Fail2ban. That can cause these kinds of issues. If it's active, disable it and see if it solves the problem.

Also, check if your VPS's IP addresses (IPv4/IPv6) are whitelisted in Fail2ban.

Btw: I know Transip. You don't have to expect a solution from them, even though they have to as a Plesk partner.

Thanks
I now disabled the plesk-modsecurity jail, I will reboot my server this evenening to test if the problem stil occurs
ip addresses were already whitelisted

Yes my plesk license is via transip.
Therefore I can't get plesk support directly and have to go to partner Transip for support.
But they say (as standard answer) they can't look on the vps, therefore can't give support

That's why I went to you guys

 

[damaverick]

There have been some updates in Plesk Obsidian 18.0.55 regarding Fail2ban:

Change Log for Plesk Obsidian

Find out about changes, additions and updates to Plesk Obsidian on an iteration to iteration basis.

docs.plesk.com

Please check if you have the plesk-modsecurity jail activated in Fail2ban. That can cause these kinds of issues. If it's active, disable it and see if it solves the problem.

Also, check if your VPS's IP addresses (IPv4/IPv6) are whitelisted in Fail2ban.

Btw: I know Transip. You don't have to expect a solution from them, even though they have to as a Plesk partner.

After a reboot the problem was back
I restored the iptables again (with input on accept and output on accept) and the problem was solved again.

So a strange issues which comes back directly after a reboot.

 

[Peter Debik]

Please check /var/log/fail2ban.log for the blocked IP address(es). It will mention the jail that adds them to iptables. From there, check why the jail is triggered.

 

[damaverick]

Please check /var/log/fail2ban.log for the blocked IP address(es). It will mention the jail that adds them to iptables. From there, check why the jail is triggered.

It's not really the triggering of a jail that is causing this problem.
After a reboot there are no blocked IP addresses in the log. Also my IP address is already in the trusted list and not showing up in the blocked list.

It's the IPtables INPUT and OUTPUT with the status drop after a reboot which is causing this problem.
After I restore a saved IP tables, where I set the INPUT and OUTPUT on accepted, my problems are solved.
The only problem now is, I have to do this after every reboot.