Resolved Network connectivity lost (iptables issue) after upgrade to latest Alma 10 kernel

[Bitpalast]

Server operating system version

Alma 10

Plesk version and microupdate number

18.0.82 with latest MU

1) A perfectly installed default setup of Plesk on Alma 10.
2) The Alma 10 kernel was upgraded to the latest version.
--> The server boots, all services are coming up, but the Plesk firewall. It cannot be started with error "status=4/NOPERMISSION".
SELinux is off, no Apparmor or other solutions are present.
iptables is empty
nft has plenty of bans from Fail2Ban

Everything looks OK including network services, but as iptables is not populated / Plesk firewall service does not start, all network traffic is blocked.

Is there any known issue regarding the upgrade to the latest Alma 10 kernel?
Server worked perfectly fine before. It's also an Alma 10 "default" installation with no special services or security software that could obviously block anything.

What could be the problem?

 

[Martin Dias]

Thank you for the report. We reproduced the issue and registered it as EXTPLESK-14623.

Cause: On AlmaLinux 10.2, the Plesk Firewall needs the kernel modules from the kernel-modules-extra package. That package is installed separately for each kernel version. Some upgrade paths, for example "dnf upgrade kernel", install a new kernel without it. After the reboot, iptables cannot load the firewall rules, and the firewall script stops with the DROP policies already set. This blocks all traffic.

To repair a server that is locked out, log in through the console (KVM, VNC or the rescue console of your provider) and run:

dnf install $(rpm -q kernel-core --qf "kernel-modules-extra-%{VERSION}-%{RELEASE}.%{ARCH} ")
systemctl restart plesk-firewall

The first command installs kernel-modules-extra for each installed kernel.

To prevent the issue on future kernel upgrades, run this once:

dnf install kernel-modules-extra-matched

This AlmaLinux meta package makes each new kernel install its matching kernel-modules-extra.

Before you reboot into a new kernel, you can check that the package is there:

rpm -q kernel-modules-extra-$(grubby --default-kernel | sed 's|/boot/vmlinuz-||')

We will fix the extension so that it does not depend on this manual step.

 

[Bitpalast]

@Martin Dias Thank you! I was no yet able to test it, but this sounds very reasonable. I'll send feedback once I have tested it.

 

[Bitpalast]

@Martin Dias First important finding is that the commands mentioned above cannot work, because the DROP rule in the INPUT chain of iptables is still in place. So we'll first run

iptables -I INPUT 1 -j ACCEPT

to make network access available (else dnf cannot download the update).

A second finding is that after restarting Plesk firewall or after reboot, the basic chains become available, but Fail2Ban is not populating iptables with the bans. It is populating nft, but that's a problem, because other scripts on the server that are interacting with iptables now need to be completely re-programmed to use nft command instead off iptables commands.

Should iptables not be auto-linked to nft so that the rules in nft are reflected in iptables?

 

[Bitpalast]

The second finding is that after restarting Plesk firewall or after reboot, the basic chains become available, but Fail2Ban is not populating iptables with the bans. It is populating nft, but that's a problem, because other scripts on the server that are interacting with iptables now need to be completely re-programmed to use nft command instead off iptables commands.

Should iptables not be auto-linked to nft so that the rules in nft are reflected in iptables?

I was wrong with that. It seems that the previous system state was already that: In Alma 10, Plesk started using nft instead of iptables, so the Fail2ban rules were previously already using nft (e.g. "action = nftables-multiport[name=apache, port="http,https,7080,7081"]").

Regarding the issue described in this thread it is solved by 1) enabling traffic to and from the server by adding the extra ACCEPT rule to iptables, 2) applying the solution by @Martin Dias.