Resolved Docker is unable to connect to the internet

[zZeepo]

Server operating system version

Ubuntu 22.04.2 LTS

Plesk version and microupdate number

Plesk Obsidian 18.0.53

Hey folks,

I'm currently migrating a project from one server to another server.
Initially I used the Plesk Migrator Extension and am now migrating the remaining changes by hand.

Unfortunately one topic is giving my quite a headache:
On my previous server I had two Docker containers, which I was able to set up quite easily without any bigger issue.
But on my new server the Docker container are all unable to connect to the internet.
Not even a simple PING to Google or other IPs is able to go through.

# docker run busybox ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes

--- 8.8.8.8 ping statistics ---
34 packets transmitted, 0 packets received, 100% packet loss

The only workaround I have found to provide an internet connection to the Docker containers is to run them with the --network=host option.
But this is causing other side effects and also seems like a dirty solution.

Because I'm currently a bit lost, I would really appreciate any advice and input to point me to a potential solution.

The server it's running on is a KVM Server of Strato.

V-Server mieten: mit Linux oder Windows | STRATO

Günstige V-Server mit viel RAM ✔ Plesk Web Admin ✔ SSL Zertifikat inklusive ✔ Unlimitierter Traffic ✔ Jetzt ab 1 € pro Monat mieten!

www.strato.de

Let me know if I can provide any more relevant information regarding this issue.

Cheers
zZeepo

These are some of the steps I have tried so far:

1. I have compared the iptables rules with my previous server.
The only difference was that the Chain FORWARD policy was DROP instead of ACCEPT, which I changed.
It didn't solve the issue tho. Here my current iptable rules:

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy DROP)
target     prot opt source               destination         
DOCKER-USER  all  --  anywhere             anywhere           
DOCKER-ISOLATION-STAGE-1  all  --  anywhere             anywhere           
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere           
ACCEPT     all  --  anywhere             anywhere           
ACCEPT     all  --  anywhere             anywhere           

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain DOCKER (1 references)
target     prot opt source               destination         

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target     prot opt source               destination         
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere           
RETURN     all  --  anywhere             anywhere           

Chain DOCKER-ISOLATION-STAGE-2 (1 references)
target     prot opt source               destination         
DROP       all  --  anywhere             anywhere           
RETURN     all  --  anywhere             anywhere           

Chain DOCKER-USER (1 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere

2. I checked for firewalls.
- ufw is "Status: inactive"
- firewalld is not installed

3. I tried to configure the DNS of the dockerd with the "/etc/docker/daemon.json" file.

{
  "dns": ["212.227.123.16", "8.8.4.4"]
}

4. I enabled IP Forwarding via "/etc/sysctl.conf"

net.ipv4.ip_forward=1
net.ipv6.conf.all.forwarding=1

5. I changed the nameserver in "/etc/resolv.conf" to Google and Cloudflare.

6. I have changed the symlink of the "/etc/resolv.conf" to "/run/systemd/resolve/resolv.conf".

 

[zZeepo]

I found a solution.
These two links proved useful to me:
[1] Docker container can not ping the outside world - iptables
[2] docker0 interface keeps losing IPv4 ipaddress · Issue #40217 · moby/moby

I noticed that the docker0 interface did not (always) have an IPv4 entry when running ip addr show docker0 (on the host).
When I ran networkctl, it looked like this:

IDX LINK    TYPE     OPERATIONAL SETUP     
  1 lo      loopback carrier     configured
  2 ens6    ether    routable    configured
  3 docker0 bridge   no-carrier  configuring


3 links listed.

Following the advice in [2] I've created the following 3 files:

/etc/systemd/network/00-dockerbridge.network

[Match]
Name=br-*

[Link]
Unmanaged=yes

/etc/systemd/network/00-dockerdefault.network

[Match]
Name=docker0

[Link]
Unmanaged=yes

/etc/systemd/network/00-veth.network

[Match]
Driver=veth

[Link]
Unmanaged=yes

Additionally I've changed the file /etc/netplan/50-cloud-init.yaml.
The entry name was changed from "*" to en* .

# This file is generated from information provided by the datasource.  Changes
# to it will not persist across an instance reboot.  To disable cloud-init's
# network configuration capabilities, write a file
# /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following:
# network: {config: disabled}
network:
    ethernets:
        all:
            dhcp4: true
            dhcp6: true
            match:
                name: en*
    renderer: networkd
    version: 2

After that I restarted systemd-networkd and docker

systemctl restart systemd-networkd
systemctl restart docker.service

Then I could see that networkctl gives different output

networkctl 
IDX LINK    TYPE     OPERATIONAL SETUP     
  1 lo      loopback carrier     configured
  2 ens6    ether    routable    configured
  3 docker0 bridge   no-carrier  unmanaged

Also ip addr show docker0 showed the IPv4 entry and I could finally connect to the internet from within a container.

docker run -it --rm busybox
/ # ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=0 ttl=58 time=0.739 ms
64 bytes from 8.8.8.8: seq=1 ttl=58 time=0.642 ms

 

[zZeepo]

One warning regarding my previous answer:
Yesterday the server was behaving slightly odd and I restarted it.
After I did that, I could not reach the server anymore. Neither ping, SSH, or any website hosted on the server worked.
The only way I managed to access it was via a VNC console provided by Strato.

I suspected that the changes I've made here might have caused the issue so I reverted them and restarted the server.
And indeed, the server was reachable again.

One thing I've noticed when I reverted the changes was, that in the file /etc/netplan/50-cloud-init.yaml the "name" entry was changed from en* to 'en'.
This was not done by me and might be the cause of the issue.

After the server was running again I've applied the same changes again which I described in the previous post but this time I changed the "name" entry in /etc/netplan/50-cloud-init.yaml to be "en*" (with quotation marks this time). I restarted the server and it was still working fine. The entry in /etc/netplan/50-cloud-init.yaml also was unchanged.

I had restarted the server before so I'm not sure what caused it to fail this time around. This also means the fix I applied here might not be the correct solution. I just want to give you guys a heads-up that the changes above can lead to issues with reachability / internet-access of your server.