Issue Docker outbound traffic gets blocked after 18.0.67 Update #3

[TheColin21]

Server operating system version

Ubuntu 22.04

Plesk version and microupdate number

18.0.67 #3

My Ubuntu 22.04 server updated Plesk from 18.0.67 U2 to 18.0.67 U3 today.
A few minutes later my monitoring emailed me about all external hosts being down.
I am running CheckMK inside a docker container.
I then noticed that all my Docker containers were now unable to communicate outside of my host. Pinging local IPs still worked, ping external IPs did not.
A reboot of the server (just for good measure) did not help.

As I did not see anything wrong with the network configuration inside the container I suspected an issue with the host.
I disabled the firewall through Plesk's UI for a minute and network communication was restored.

I then reenabled the firewall but set the "Systemrichtlinie für die Weiterleitung des Datenverkehrs" (german installation, the english translation would be "system policy for forwarding traffic") to allow.

This workaround does allow the containers to work correctly again but decreases security.

The changelog for 18.0.67 U3 did not reveal anything firewall-related.
How can I correct this?

 

[ArchieCG]

Just ran into this issue today. All dockers got blocked outside connection by this update.

Had to turn off my firewall to make them work again.

Looking for a fix in this situation.

 

[Master_rg]

Debian 12.9 with 18.0.67 #3

We got several servers with this configuration, all Docker Containers have the same connectivity issues mentioned by TheColin21.

 

[Sebahat.hadzhi]

Thank you all for the report. I will discuss it with our team and update you shortly. For the time being, please apply @TheColin21's workaround for setting "System policy for traffic forwarding" to allowed.

 

[KaiE]

Same here, since this morgning i was fighting the firewall, and now found this post here.

An addition: it seems with the workaround from @TheColin21 the nginx proxy rules aren't working anymore. If I turn off the Firewall the Containers are able to access and be accessed normally, if I turn on the firewall and allow the traffic forwarding, the containers aren't reachable via Domain and proxy rule.

 

[TheColin21]

Same here, since this morgning i was fighting the firewall, and now found this post here.

An addition: it seems with the workaround from @TheColin21 the nginx proxy rules aren't working anymore. If I turn off the Firewall the Containers are able to access and be accessed normally, if I turn on the firewall and allow the traffic forwarding, the containers aren't reachable via Domain and proxy rule.

I have several containers that I access via nginx proxy rules, they all work.
Maybe something you did during your fight caused this additional issue

 

[KaiE]

@TheColin21
Ha, that's very well possible
I tried to make sure to document any change and revert it after I found this thread here, but I'll have another look as soon as there's a fix/permanent solution available.

 

[Sebahat.hadzhi]

I discussed the case with our team and no changes were applied on the Docker extension and the Firewall during the last update. For what is worth, during a test I experienced the same behavior for the outbound connections from a container on a Ubuntu 22.04 server running Plesk Obsidian 18.0.65 Update #2. While I can't ping 8.8.8.8 from the container, I can from the server. It seems like there are other people complaining about the issue on Dokcer's forum and GitHub. I will try rolling back the Docker version on the test servers and I will let you know.

 

[TheColin21]

I'm surprised you encountered it on #2 - that updated worked flawlessly for me.

 

[KaiE]

I'm surprised as well!
Checking the Plesk Action Log I can't see any Plesk Update at the point when the error symptoms started, and at that time the server was at Update #2.

But: at exactly the point in time two things are in the log:

First: a plesk admin licence renewal, done twice within two seconds.

Second: Upgrade extension

Extension Id: nodejs
Extension Name: Node.js Toolkit
Extension Release: 3990 → 2404
Extension Version: 2.3.15 → 2.4.0

Both seem to have no connection to Docker or Firewall-Behaviour, but it might be worth mentioning!
Was it maybe a previously updated setting, and e.g. one of the two mentioned things restarted a service which wasn't restarted before?

Just my two cents...

 

[Yurij]

I have a similar problem with the docker after the Plesk update in the morning: dial tcp: lookup mysite.com: i/o timeout

 

[ArchieCG]

Update #2 had no issues on our side.

Only today after the #3 update all went down.

 

[ksau]

Same here. Changing the firewall rule "System policy for traffic forwarding" from "Deny" to "Allow" worked for me to resolve the issue. Not sure wether one wants to allow that for everything.

 

[christian.r.r]

Same behavior here on two different Plesk Servers, one without Update on #2 and one up-to-date on #3. Outbound traffic from Docker containers suddenly stopped working from yesterday to today.

The workaround with setting the system policy works, but is far from an ideal solution from a security perspective. Anyway, thanks to @ksau for suggesting it!

@Sebahat.hadzhi Please supply a fix!

​

 

[Sebahat.hadzhi]

@christian.r.r I have already opened a case for further investigation by our engineers and will provide details as soon as possible.

 

[JCD]

I ran into the same issue, i disabled the firewall and found your thread afterwards.

To create some urgency in the highly dockerized world

+1

 

[Sebahat.hadzhi]

Everyone, thank you for your patience. The case was investigated by our engineers and they determined the root cause to be Docker adding new firewall forwarding rules to the end after DROP:

-A FORWARD -j DROP
-A FORWARD -i br-c5a345a92ad7 -j ACCEPT
-A FORWARD -i docker0 -j ACCEPT
-A FORWARD -i br-353937b03a8c -j ACCEPT

The workaround is:

1. Connect to the Plesk server via SSH.
2. Block installation of this version:
   
   

   vi /etc/apt/preferences.d/docker-block

   Ubuntu:
   

   Package: docker-ce
   Pin: version 5:28.0.0-1~ubuntu.22.04~jammy
   Pin-Priority: -1
   
   Package: docker-ce-cli
   Pin: version 5:28.0.0-1~ubuntu.22.04~jammy
   Pin-Priority: -1

   Debian:
   

   Package: docker-ce
   Pin: version 5:28.0.0-1~debian.12~bookworm
   Pin-Priority: -1
   
   Package: docker-ce-cli
   Pin: version 5:28.0.0-1~debian.12~bookworm
   Pin-Priority: -1

3. Remove installed version:
   

   # sudo apt remove docker-ce docker-ce-cli
   # sudo apt autoremove
   # sudo apt install docker-ce

Afterward, the latest available version 27.5.1 will be installed. I hope that helps!
Let me know if instructions are needed for another OS/version, please.

 

[TheColin21]

Everyone, thank you for your patience. The case was investigated by our engineers and they determined the root cause to be Docker adding new firewall forwarding rules to the end after DROP:



The workaround is:

1. Connect to the Plesk server via SSH.
2. Block installation of this version:
   
   
   Ubuntu:
   
   Debian:
3. Remove installed version:

Afterward, the latest available version 27.5.1 will be installed. I hope that helps!
Let me know if instructions are needed for another OS/version, please.

Just to be sure: Are we sure this will be fixed in the next docker release? If not, the problem will just reappear.

 

[devnull]

Everyone, thank you for your patience. The case was investigated by our engineers and they determined the root cause to be Docker adding new firewall forwarding rules to the end after DROP:



The workaround is:

1. Connect to the Plesk server via SSH.
2. Block installation of this version:
   
   
   Ubuntu:
   
   Debian:
3. Remove installed version:

Afterward, the latest available version 27.5.1 will be installed. I hope that helps!
Let me know if instructions are needed for another OS/version, please.

Thank you but this doesn’t sound like a real fix. When will there be an official and reliable fix for this solution as there are many running production stuff in containers.

 

[christian.r.r]

Everyone, thank you for your patience. The case was investigated by our engineers and they determined the root cause to be Docker adding new firewall forwarding rules to the end after DROP:



The workaround is:

1. Connect to the Plesk server via SSH.
2. Block installation of this version:
   
   
   Ubuntu:
   
   Debian:
3. Remove installed version:

Afterward, the latest available version 27.5.1 will be installed. I hope that helps!
Let me know if instructions are needed for another OS/version, please.

Wouldn't we loose our docker containers and configs if we uninstall the current docker engine? Or do I miss something here?