Issue Firewall suddenly took Plesk down overnight

[Tim_Wakeling]

Server operating system version

AlmaLinux 8.8

Plesk version and microupdate number

18.0.53

Something very odd happened last night on my server. About 4 am, when nothing was due to run (the backup completed at 1.25 am), all sites went down.

I could not log into Plesk, but I did have the CLI. nginx crashed when I tried to restart it, and plesk repair all -y also failed at the stage of reconfiguring the domains.

Then I tried pinging to 8.8.8.8 and other IPs, and all failed.

However now for some reason (probably due to the repair) I was able to log back into Plesk, even though all the websites were still down.

So from there I turned off the firewall, and everything returned.

With some trepidation, I turned the firewall back on, and everything was still fine. So turning it off and on again without any rule changes fixed it.

My firewall rules were (and are):

• DHCP, Samba, DNS and IPv6 denied to all
• SMTP, POP3, IMAP allowed to all
• Inbound pings, FTP, SSH and Plesk Installer all allowed from my IP only, and one other IP belonging to a specific client

And they did not change last night as far as I know; I last tweaked them a couple of weeks ago.

Nothing recorded in /var/log/sw-cp-server/error_log around the incident time (or since), but in /var/log/fail2ban.log around 4am I can see this:

2023-07-18 04:06:16,064 fail2ban.actions [817]: NOTICE [recidive] Unban 208.38.235.61
2023-07-18 04:06:16,094 fail2ban.utils [817]: ERROR 7f59404b4870 -- exec: iptables -w -n -L INPUT | grep -q 'f2b-recidive[ \t]'
2023-07-18 04:06:16,095 fail2ban.utils [817]: ERROR 7f59404b4870 -- returned 1
2023-07-18 04:06:16,095 fail2ban.CommandAction [817]: ERROR Invariant check failed. Unban is impossible.
2023-07-18 04:06:16,096 fail2ban.actions [817]: ERROR Failed to execute unban jail 'recidive' action 'iptables-allports' info 'ActionInfo({'ip': '208.38.235.61', 'fid': <function <lambda> at 0x7f5951eaa2d0>, 'family': 'inet4', 'raw-ticket': <function <lambda> at 0x7f5951eaa8d0>})': Error unbanning 208.38.235.61

Nothing else in that log until 3 hours later after those lines.

Looks like an attempt perhaps by the owner of 208.38.235.61 to get themselves in? I don't know that IP.

Any idea what might have happened, why the firewall might have fallen over entirely as a result, and how I can stop that happening again?

Appreciate it's a tricky mystery!

Thanks so much

Tim

 

[Peter Debik]

Please check if you have the firewalld service active. If so, please disable it.

 

[Tim_Wakeling]

Just tried that and it took everything down! Had to restart the server manually from the VPS control panel. Now it's all back up.

OK, I think the fail2ban thing was a red herring. I've tracked down the exact time things went wrong, and it was about 3.20 am, just when the following updates were applied:

The following packages were successfully updated:
- NetworkManager 1:1.40.16-3.el8_8.alma from baseos repo (previous version: 1:1.40.16-1.el8 from baseos repo)
- NetworkManager-libnm 1:1.40.16-3.el8_8.alma from baseos repo (previous version: 1:1.40.16-1.el8 from baseos repo)
- NetworkManager-team 1:1.40.16-3.el8_8.alma from baseos repo (previous version: 1:1.40.16-1.el8 from baseos repo)
- NetworkManager-tui 1:1.40.16-3.el8_8.alma from baseos repo (previous version: 1:1.40.16-1.el8 from baseos repo)
- kexec-tools 2.0.25-5.el8_8.1.alma from baseos repo (previous version: 2.0.25-5.el8 from baseos repo)
- microcode_ctl 4:20220809-2.20230214.1.el8_8.alma from baseos repo (previous version: 4:20220809-2.el8 from baseos repo)
- xfsprogs 5.0.0-11.el8_8.alma from baseos repo (previous version: 5.0.0-10.el8 from baseos repo)

It seems one of these must have tripped up the firewall somehow...

 

[Peter Debik]

firewalld may have removed all entries from iptables when disabled. However, running both services can create the "Invariant check failed" error posted above.

 

[Tim_Wakeling]

By "both services", do you mean firewalld and fail2ban?

 

[Peter Debik]

firewalld along with iptables.

 

[Tim_Wakeling]

I confess I didn't know about iptables. I used the Firewall from the Plesk interface, and it seemed to work as advertised.

What does iptables do, and should I remove it if I'm using the Firewall in the Plesk interface? How would I do that?

Thanks!

 

[Peter Debik]

Please do not remove iptables. iptables and ipset are both used by the Plesk firewall extension.

 

[Tim_Wakeling]

Hi again ... another NetworkManager update occurred this morning and once again, all websites went down, just like above. I restarted the firewall and everything was back up.

My host advised me to disable NetworkManager and revert to using network-scripts, which I have done, but realising that network-scripts is deprecated, I imagine I am going to have to go back to NetworkManager at some point.

If so, then I will need to understand why each NetworkManager update breaks the firewall and causes "Invariant check failed" errors until the firewall rules are turned off and on again. Can you help me work it out? You mentioned that it might be removing all entries from iptables, but I don't quite understand that and am not sure what to check.

Thank you so much!

 

[Peter Debik]

When you are using the Plesk Firewall extension you can safely disable Firewalld. Both at the same time will create conflicts. Firewalld is not needed if you have the Plesk Firewall extension.

 

[manos]

firewalld along with iptables.

Peter I think you mean firewalld along with plesk-firewall service that both write to iptables

Some providers install correctly plesk with almalinux and firewalld is disabled while all needed ports are already at iptables
via plesk firewall, at others firewalld is enabled with open only ssh dhcp and cokpit and need to open manually the others.
At these configurations if you disabled firewalld you will be locked out from ssh as iptable entries will be erased.

Btw at some plesk servers the service is named psa-firewall at others plesk-firewall, same versions, why this difference?

 

[manos]

When you are using the Plesk Firewall extension you can safely disable Firewalld. Both at the same time will create conflicts. Firewalld is not needed if you have the Plesk Firewall extension.

As i wrote at same thread it is not safe, can be locked out from server.

 

[Tim_Wakeling]

Hi Manos - thanks for the heads-up. How would I go about checking whether firewalld is configured in the way you have highlighted, so as to know whether it's safe to disable?

Thanks!

 

[Tim_Wakeling]

I just tried checking the status of plesk-firewall and firewalld, and got the following output:

$ systemctl status plesk-firewall
● plesk-firewall.service - Plesk firewall rules
Loaded: loaded (/usr/lib/systemd/system/plesk-firewall.service; enabled; vendor preset: disabled)
Active: active (exited) since Mon 2023-08-28 07:48:59 BST; 12min ago
Process: 1113 ExecStart=/bin/bash /usr/local/psa/var/modules/firewall/firewall-active.sh (code=exited, status=0/SUCCESS)
Main PID: 1113 (code=exited, status=0/SUCCESS)
Tasks: 0 (limit: 74820)
Memory: 0B
CGroup: /system.slice/plesk-firewall.service

$ systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2023-08-28 07:48:47 BST; 13min ago
Docs: man:firewalld(1)
Main PID: 692 (firewalld)
Tasks: 2 (limit: 74820)
Memory: 43.3M
CGroup: /system.slice/firewalld.service
└─692 /usr/libexec/platform-python -s /usr/sbin/firewalld --nofork --nopid

Plesk Firewall shows as "active (exited)" and firewalld shows as "active (running)". Is that normal? Does this mean that the Plesk Firewall is not doing anything, regardless of what rules I set in the admin UI? In the admin UI, Plesk Firewall seems to be enabled and has a set of custom rules that I configured.

If I try to restart firewalld the server crashes and I have to restart the whole server to bring it back up. I imagine if I stopped it the same would happen.

 

[manos]

Hi Manos - thanks for the heads-up. How would I go about checking whether firewalld is configured in the way you have highlighted, so as to know whether it's safe to disable?

Thanks!

Hi Tim

At that almalinux/plesk machine if i remember well i did this:

iptables -S had the open ports (this is tables not the service, obviously the ports added by firewalld)

Then before disabling firewallD i installed iptables as service at almalinux in order
entries of open ports to not be deleted after stopping firewallD

dnf install iptables-services
systemctl start iptables
systemctl start ip6tables
systemctl enable iptables
systemctl enable ip6tables

check now again with iptables -S if entries (especiall SSH are open there)

systemctl stop firewalld
systemctl disable firewalld
systemctl mask --now firewalld

Restarted and could enter SSH as entries remained

Then stopped iptables.service , enabled plesk firewall and diabled iptables.service

 

[Peter Debik]

Again: Running both at the same time can create conflicts and issues. The official Plesk advice is to disable firewalld when you are using the Plesk Firewall service.

Plesk for Linux and firewalld Compatibility

firewalld is a firewall management tool for Linux operating systems. This section explains how you can use it to open...

docs.plesk.com

As i wrote at same thread it is not safe, can be locked out from server.

Under which circumstances will that occur?

 

[manos]

Again: Running both at the same time can create conflicts and issues. The official Plesk advice is to disable firewalld when you are using the Plesk Firewall service.

Plesk for Linux and firewalld Compatibility

firewalld is a firewall management tool for Linux operating systems. This section explains how you can use it to open...

docs.plesk.com

Under which circumstances will that occur?

Hi Peter thanks for coming back
Right when disabling firewallD locked out, obviously it deletes the 3 default entries at iptables including ssh
It seems that depends on how plesk installed by the provider, at main vps/dedicated provider i use
never have proble, just activating plesk firewall and all ok.
At the other one while same OS Almalnux 8 and latest Plesk have the problem mentioned.

Btw, do nto know if you saw at other thread, why at some installations service is names psa-firewall and at other plesk-firewall?

 

[Tim_Wakeling]

Thank you Manos! I followed your instructions, installed the iptables service (it was not installed) and was able to disable firewalld.

Now its status shows as:

$ systemctl status firewalld
● firewalld.service
Loaded: masked (Reason: Unit firewalld.service is masked.)
Active: inactive (dead)

Plesk Firewall is still active (exited) as before:

$ systemctl status plesk-firewall
● plesk-firewall.service - Plesk firewall rules
Loaded: loaded (/usr/lib/systemd/system/plesk-firewall.service; enabled; vendor preset: disabled)
Active: active (exited) since Mon 2023-08-28 08:26:10 BST; 1min 31s ago
Process: 1085 ExecStart=/bin/bash /usr/local/psa/var/modules/firewall/firewall-active.sh (code=exited, status=0/SUCCESS)
Main PID: 1085 (code=exited, status=0/SUCCESS)
Tasks: 0 (limit: 74820)
Memory: 0B
CGroup: /system.slice/plesk-firewall.service

This now seems good, and everything is working. However I didn't follow your last instruction to disable and remove iptables, because what Peter was saying earlier made me think it could be problematic to do that. At the moment on my server the iptables service is running and Plesk Firewall is running, but firewalld is not.

The command iptables -S now shows a full list of rules that seem to be exactly the same as the ones in Plesk Firewall's UI that I set up (I won't copy and paste those here.) So should I now just leave it be?

 

[manos]

Thank you Manos! I followed your instructions, installed the iptables service (it was not installed) and was able to disable firewalld.

Now its status shows as:

$ systemctl status firewalld
● firewalld.service
Loaded: masked (Reason: Unit firewalld.service is masked.)
Active: inactive (dead)

Plesk Firewall is still active (exited) as before:

$ systemctl status plesk-firewall
● plesk-firewall.service - Plesk firewall rules
Loaded: loaded (/usr/lib/systemd/system/plesk-firewall.service; enabled; vendor preset: disabled)
Active: active (exited) since Mon 2023-08-28 08:26:10 BST; 1min 31s ago
Process: 1085 ExecStart=/bin/bash /usr/local/psa/var/modules/firewall/firewall-active.sh (code=exited, status=0/SUCCESS)
Main PID: 1085 (code=exited, status=0/SUCCESS)
Tasks: 0 (limit: 74820)
Memory: 0B
CGroup: /system.slice/plesk-firewall.service

This now seems good, and everything is working. However I didn't follow your last instruction to disable and remove iptables, because what Peter was saying earlier made me think it could be problematic to do that. At the moment on my server the iptables service is running and Plesk Firewall is running, but firewalld is not.

The command iptables -S now shows a full list of rules that seem to be exactly the same as the ones in Plesk Firewall's UI that I set up (I won't copy and paste those here.) So should I now just leave it be?

FirewallD as Peter mentioned should be correctly disabled (or masked)

plesk-firewall service is correctly enabled

As for iptables, did not find any documentation on this, but as i tried myself
realized that iptables and iptables.service at least at Almalinux are different things.
Almalinux 8 has iptables (on a fresh install with iptables -S you will see the entries)
but not iptables.service as you mentioned also.
I did same things as you and disabled iptables.service , still iptables -S shows the plesk firewall opened ports
so my config at this machine is firewalld disabled, plesk-firewall.service enabled, iptables.service disabled

Hope Peter can help us to clear meanings eh?

 

[Peter Debik]

You'll want to keep iptables, because that is where the firewall rules act. iptables is the command to add or delete ruls and chains. It does not need the service. But: The iptables.service is needed to automatically load saved rulesets on boot and also unload them on shutdown. In my opinion you should keep both.