Resolved Firewall disabled

[Kais72]

Server operating system version

Ubuntu 22.04.2 LTS

Plesk version and microupdate number

Plesk Obsidian Versión 18.0.52

Hi all,

I'm having problems enabling Plesk Firewall.
It had been working perfectly the last two years, but now it's disabled and can't find a way to enable it again.
I'm not sure where this issue come from, but: a few days ago I removed a custom rule (for varnish) -> Firewall told me it had to exec a script -> the I get an error saying: "Command '['/usr/local/psa/var/modules/firewall/firewall-emergency.sh']' timed out after 5 seconds"

I've tried to uninstall/install back with no luck.

Any help is welcome.
Thanks.

 

[Peter Debik]

Is the service running?
# service psa-firewall restart

 

[Kais72]

Hi Peter,
Nope: "Unit psa-firewall.service not found"

 

[Peter Debik]

The first step to resolve this is probably following the short guide in https://support.plesk.com/hc/en-us/articles/12387716023063

If and only if that does not work for you - after you have attempted these steps - you can try to run these two commands:

# /usr/local/psa/bin/modules/firewall/settings --enable
# /usr/local/psa/bin/modules/firewall/settings --confirm

 

[Kais72]

Thanks for your help Peter,
Unfortunately, following the short guide didn't help, got the same error message.

So I ran those two commands:

• Enable command --> "Firewall rules management was enabled. To save your changes, run the --confirm command within 60 second(s)."
• Confirm command (10 seconds later, or so) --> "Too late to confirm: no rules activation process. Unable to confirm firewall rules management."

 

[Liwindo]

Looks like your issue and mine are connected with each other since I get the same output on OS. Could you please verify if your iptables gets also flushed after a few minutes when the command was executed via

iptables -L

 

[amba1980]

I have the same problem. Fresh install Obsidian / Ubuntu 22.04 LTS. Confirming the rules times out for no apparent reason.

I have removed and reinstalled Firewall through the UI and using ssh, same effect: Can't enable the firewall.

 

[amba1980]

Here the message from the UI, suggesting there never was a good configuration (this is a fresh install!)

• I did not receive connectivity confirmation after applying new firewall configuration, then same happened after I reverted to previous configuration. This means that both new and previous configurations were bad. Emergency rollback to configuration without rules was performed. Firewall is now disabled. Fix your rules and try again.

 

[Peter Debik]

Meanwhile this has been filed as a product issue here, ID PPS-14320. Please contact our support staff mentioning this ID if your case is urgent.

 

[Peter Debik]

Guys, we're looking for users who experience this issue and are willing to provide SSH access to their systems for our engineers. We cannot reproduce the issue here and we don't have any support tickets with SSH access allowed on it, so currently we cannot find out the cause as all on of our own test systems the issue does not occur.

So, please, if possible open a support ticket and give support SSH access by the SSH support extension (free in the extensions section). If you are using a reseller license you can still create a support ticket using the 30-day free trial period for a support subscription:

https://support.plesk.com/hc/en-us/articles/12388090147095-How-to-get-support-directly-from-Plesk-

 

[Kais72]

Thanks Peter,
I'll try to open a support ticket and give SSH access today.

 

[danami]

@Peter Debik I've noticed this error and some others.

1. Our Warden extension calling --enable and --confirm over a web/app server worker process no longer works any more as it complains that it didn't get SSH confirmation even though it's not being called over SSH (it's using a pm_LongTask_Task). This worked fine for the last 3 years with the old firewall extension.

You can see that it should work by using: /usr/local/psa/admin/bin/modules/firewall/rules --help

--confirm Commit activation of the new rules. Should be invoked from a new SSH session or web/app server worker process to ensure an existing network connection is not re-used.

2. As others have noticed the timeout (PHP_SAFEACT_CONFIRM_INTERVAL) isn't working any more.:

[[email protected] ~]# /usr/local/psa/bin/modules/firewall/settings --enable
[2023-04-07 20:17:13.791] 1083629:6430dcb9c068d DEBUG [util_exec] Starting with task-manager: /usr/local/psa/admin/bin/php /usr/local/psa/admin/plib/scripts/task-async-executor.php -task-id 183
Firewall rules management was enabled. To save your changes, run the --confirm command within 60 second(s).

Wait over 15 seconds and apply the confirm in a new SSH terminal:

[[email protected] ~]# /usr/local/psa/bin/modules/firewall/settings --confirm
[2023-04-07 20:17:39.913] 1084617:6430dcd3de88b DEBUG [util_exec] [946e8a2b300f36e4abc60286acefcfb5-0] Starting: osdetect
[2023-04-07 20:17:39.923] 1084617:6430dcd3de88b DEBUG [util_exec] [946e8a2b300f36e4abc60286acefcfb5-0] Finished in 0.00649s, Error code: 0
[2023-04-07 20:17:39.926] 1084617:6430dcd3de88b DEBUG [extension/firewall] [6430dcd3e23a5] Starting: '/usr/local/psa/admin/bin/modules/firewall/rules'  '--confirm'
[2023-04-07 20:17:39.967] 1084617:6430dcd3de88b DEBUG [extension/firewall] [6430dcd3e23a5] Finished in 0.04024s, Error code: 1
Too late to confirm: new rules were rolled back
Unable to confirm the firewall rules.

3. Firewall rules are not reverted properly. If confirmation fails then it doesn't apply the old rules. It just disables the firewall completely:

[2023-04-07 20:22:50.469] 1084763:6430ddec14bf8 ERR [panel] Long task executor: id=184 completed with error: I did not receive connectivity confirmation after applying new firewall configuration, then same happened after I reverted to previous configuration. This means that both new and previous configurations were bad. Emergency rollback to configuration without rules was performed. Firewall is now disabled. Fix your rules and try again.

# iptables -L
# Warning: iptables-legacy tables present, use iptables-legacy to see them
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

 

[Peter Debik]

@danami We have this issue as "expedited" in our records. Engineers are aware of it and are already working on it. It is linked with "too long" Nginx restart times. Why these occur is not clear yet. They do not occur on all installations, only on some. They do not occur on any of our own test servers. It is currently under investigation. The urgency is fully understood here, but I do not have an ETA for a fix yet.

Personally, but this is not official Plesk advice, I'd suggest you check the duration that an "# nginx -t" takes. If this takes more than 5 seconds, I think it is likely that the resolvers in /etc/resolv.conf cause a long wait. You can try to switch them to other public resolvers just for a test and then try again if it lowers the Nginx config test duration.

 

[danami]

Thanks for the update Peter. It happens on all my test VMs here (running nginx -t is instant). It also happens on all our client servers too as I can see it happen when I do Warden installs for our clients.

 

[danami]

@Peter Debik One last note that all my test VMs only have 1 or 2 domains in the panel and they are all using Google public DNS (8.8.8.8 and 8.8.4.4). So I can't see this being an nginx timeout issue (as I mentioned running nginx -t is instant).

 

[VNick]

@danami, I'm sure other Plesk devs already communicated this to the Warden devs, but I'll duplicate this information here, just in case.

You can see that it should work by using: /usr/local/psa/admin/bin/modules/firewall/rules --help

--confirm Commit activation of the new rules. Should be invoked from a new SSH session or web/app server worker process to ensure an existing network connection is not re-used.

This is an internal utility. Nobody should use it directly or pay attention to whatever is written in its help, except for the Firewall extension developers. That said, it may indeed be useful to gain insight in how the Firewall works. However, the provided interpretation of the help message is incorrect. "new" in the sentence above relates BOTH to SSH sessions and web/app server worker processes. Key point here is "to ensure an existing network connection is not re-used".

This worked fine for the last 3 years with the old firewall extension.

The same constraint existed before. It might not have been enforced as strictly, but it doesn't mean one is allowed to break it.

As others have noticed the timeout ... isn't working any more.

This is related to CLI (settings utility) only, and is not something that others in the thread initially encountered. That said, we're aware of this, thank you.

Firewall rules are not reverted properly. If confirmation fails then it doesn't apply the old rules.

Rules are actually rolled back to the original configuration. However, if it is not confirmed (after the rollback), emergency rollback to the configuration w/o rules is done.

 

[danami]

@VNick We aren't calling `/usr/local/psa/admin/bin/modules/firewall/rules` directly. We are just following the debug information (It gets called when calling --confirm.). Example you can test:

\pm_ApiCli::call('extension', array('--call', 'firewall', '--confirm')

I still don't understand why this was changed. Now there is no way for a extensions to open a port on the firewall without having to rely on hacks by adding iptables rules manually. Everything worked fine before.

 

[VNick]

Applying firewall configuration should be an interactive operation. Otherwise one risks completely cutting off access to the server for the user.

After adding your customizations show a notification in Plesk that suggests the user to go to the Firewall page, review the rules, and apply the changed configuration. Do not apply the configuration automatically.

 

[danami]

> After adding your customizations show a notification in Plesk that suggests the user to go to the Firewall page, review the rules, and apply the changed configuration. Do not apply the configuration automatically.

That's not ideal. Most users are non-technical and would have trouble with that. Also the majority of service providers install and configure Plesk extensions automatically using tools like Ansible. You can't be asking users to manually go to a page to apply settings as the server needs to be fully working on automated deployment.

Idealy in pre-install.php you should be able to do:

\pm_ApiCli::call('extension', array('--call', 'firewall', '--set-rule', '-name', 'Test service', '-direction', 'output', '-action', 'allow', '-ports', '2703/tcp'), \pm_ApiCli::RESULT_FULL))
\pm_ApiCli::call('extension', array('--call', 'firewall', '--apply'), \pm_ApiCli::RESULT_FULL)
\pm_ApiCli::call('extension', array('--call', 'firewall', '--confirm', \pm_ApiCli::RESULT_FULL)

Then in pre-uninstall.php you would remove any added rules.

 

[VNick]

To the original posters: Firewall 2.0.2 was released. It includes a number of improvements to address activation issues you've encountered. See also its changelog for instructions on increasing confirmation timeout in case simple extension update will not resolve the issue in your case. CLI timeout issue was addressed as well.

Thank you for reaching out to support and allowing us to investigate these issues.

The root cause in each case was not in the Firewall itself, but in some other issue on the server - typically an excessively slow service restart or slow iptables operation. It's best to resolve them as well.