Issue Docker containers don't start after plesk/extension upgrades

[kassi]

Server operating system version

Ubuntu 22.04.3 LTS

Plesk version and microupdate number

18.0.57 #5

Last night the docker related packages have been updated on the system automatically (containerd.io docker-compose-plugin docker-ce-cli docker-ce).
This morning no container was running, although every container should restart automatically.

I can see in the syslog around 01:17, that containerd is being stopped and started. However there are a lot of errors listed.

If I try to start a container manually, it fails with the message:

{"message":"failed to create task for container: task bb5f956df7a59461df6ae8dfb0f110e737c75dc73fea1b1d5174d8341bcb7310: already exists"}

Since I had comparable problems last time, where 1 out of 4 containers won't start and it was a manual hassle to get it all running again, I'd like to understand:
1. How can I fix it so that the containers will start again?
2. How can I prevent this happening again on the next package upgrade?

Thank you for helping.

Here's an excerpt of the egrep '(containerd|docker)' /var/log/syslog.1 | grep 'Feb 4'. Full log from this day (docker|containerd) attached.

Feb  4 01:17:18 myserver systemd[1]: Stopping containerd container runtime...
Feb  4 01:17:18 myserver systemd[1]: containerd.service: Deactivated successfully.
Feb  4 01:17:18 myserver systemd[1]: containerd.service: Unit process 3400 (containerd-shim) remains running after unit stopped.
Feb  4 01:17:18 myserver systemd[1]: containerd.service: Unit process 3452 (containerd-shim) remains running after unit stopped.
...
Feb  4 01:17:18 myserver dockerd[1477]: time="2024-02-04T01:17:18.909092432+01:00" level=error msg="Failed to get event" error="rpc error: code = Unavailable desc = error reading from server: EOF" module=libcontainerd namespace=moby
Feb  4 01:17:18 myserver dockerd[1477]: time="2024-02-04T01:17:18.909254276+01:00" level=error msg="Failed to get event" error="rpc error: code = Unavailable desc = error reading from server: EOF" module=libcontainerd namespace=plugins.moby
Feb  4 01:17:18 myserver dockerd[1477]: time="2024-02-04T01:17:18.909878626+01:00" level=info msg="Waiting for containerd to be ready to restart event processing" module=libcontainerd namespace=plugins.moby
Feb  4 01:17:18 myserver dockerd[1477]: time="2024-02-04T01:17:18.909886941+01:00" level=info msg="Waiting for containerd to be ready to restart event processing" module=libcontainerd namespace=moby
Feb  4 01:17:20 myserver dockerd[1477]: time="2024-02-04T01:17:20.322437303+01:00" level=warning msg="Health check for container 05dae063affeed5dafeca0280b3528647d016758e3ad4c274b5a6726185226a3 error: connection error: desc = \"transport: Error while dialing: dial unix:///run/containerd/containerd.sock: timeout\": unavailable"
Feb  4 01:17:21 myserver systemd[1]: Stopped containerd container runtime.
Feb  4 01:17:21 myserver systemd[1]: containerd.service: Found left-over process 3400 (containerd-shim) in control group while starting unit. Ignoring.
...
Feb  4 01:17:25 myserver dockerd[1477]: time="2024-02-04T01:17:25.323687421+01:00" level=warning msg="Health check for container 05dae063affeed5dafeca0280b3528647d016758e3ad4c274b5a6726185226a3 error: connection error: desc = \"transport: Error while dialing: dial unix:///run/containerd/containerd.sock: timeout\": unavailable"
Feb  4 01:17:26 myserver containerd[346095]: time="2024-02-04T01:17:26+01:00" level=warning msg="containerd config version `1` has been deprecated and will be removed in containerd v2.0, please switch to version `2`, see https://github.com/containerd/containerd/blob/main/docs/PLUGINS.md#version-header"
Feb  4 01:17:26 myserver containerd[346095]: time="2024-02-04T01:17:26.398395667+01:00" level=info msg="starting containerd" revision=ae07eda36dd25f8a1b98dfbf587313b99c0190bb version=1.6.28
Feb  4 01:17:26 myserver containerd[346095]: time="2024-02-04T01:17:26.422873554+01:00" level=info msg="loading plugin \"io.containerd.content.v1.content\"..." type=io.containerd.content.v1
Feb  4 01:17:26 myserver containerd[346095]: time="2024-02-04T01:17:26.422963032+01:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.aufs\"..." type=io.containerd.snapshotter.v1
Feb  4 01:17:26 myserver containerd[346095]: time="2024-02-04T01:17:26.425584596+01:00" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.aufs\"..." error="aufs is not supported (modprobe aufs failed: exit status 1 \"modprobe: FATAL: Module aufs not found in directory /lib/modules/5.2.0\\n\"): skip plugin" type=io.containerd.snapshotter.v1
...
Feb  4 01:17:26 myserver containerd[346095]: time="2024-02-04T01:17:26.516440761+01:00" level=error msg="failed to initialize a tracing processor \"otlp\"" error="no OpenTelemetry endpoint: skip plugin"
Feb  4 01:17:26 myserver containerd[346095]: time="2024-02-04T01:17:26.516826024+01:00" level=info msg=serving... address=/run/containerd/containerd.sock.ttrpc
Feb  4 01:17:26 myserver containerd[346095]: time="2024-02-04T01:17:26.516887569+01:00" level=info msg=serving... address=/run/containerd/containerd.sock
Feb  4 01:17:26 myserver containerd[346095]: time="2024-02-04T01:17:26.516952991+01:00" level=info msg="containerd successfully booted in 0.121445s"
Feb  4 01:17:26 myserver systemd[1]: Started containerd container runtime.
Feb  4 01:18:08 myserver dockerd[1477]: time="2024-02-04T01:18:08.730475718+01:00" level=info msg="Processing signal 'terminated'"
Feb  4 01:18:19 myserver dockerd[1477]: time="2024-02-04T01:18:19.146783303+01:00" level=info msg="Container failed to exit within 10s of signal 15 - using the force" container=2616af3cf113c404d29d34f239816b778a1afc27907b0708ce3bb7ce37b77ea2
...
Feb  4 01:18:23 myserver dockerd[1477]: time="2024-02-04T01:18:23.916394785+01:00" level=error msg="Force shutdown daemon"
Feb  4 01:18:23 myserver dockerd[1477]: time="2024-02-04T01:18:23.921558984+01:00" level=info msg="Daemon shutdown complete"
Feb  4 01:18:23 myserver dockerd[1477]: time="2024-02-04T01:18:23.921727029+01:00" level=info msg="stopping event stream following graceful shutdown" error="context canceled" module=libcontainerd namespace=plugins.moby
Feb  4 01:18:23 myserver dockerd[1477]: time="2024-02-04T01:18:23.921769018+01:00" level=info msg="stopping event stream following graceful shutdown" error="context canceled" module=libcontainerd namespace=moby
Feb  4 01:18:24 myserver systemd[1]: docker.service: Deactivated successfully.
Feb  4 01:18:29 myserver systemd[1]: docker.socket: Deactivated successfully.
Feb  4 01:18:36 myserver dockerd[347660]: time="2024-02-04T01:18:36.344623745+01:00" level=info msg="Starting up"
Feb  4 01:18:36 myserver dockerd[347660]: time="2024-02-04T01:18:36.354365744+01:00" level=info msg="detected 127.0.0.53 nameserver, assuming systemd-resolved, so using resolv.conf: /run/systemd/resolve/resolv.conf"
Feb  4 01:18:38 myserver systemd[1]: var-lib-docker-overlay2-metacopy\x2dcheck1477200603-merged.mount: Deactivated successfully.
Feb  4 01:18:38 myserver dockerd[347660]: time="2024-02-04T01:18:38.446822596+01:00" level=info msg="[graphdriver] using prior storage driver: overlay2"
Feb  4 01:18:38 myserver dockerd[347660]: time="2024-02-04T01:18:38.525768941+01:00" level=info msg="Loading containers: start."
Feb  4 01:18:38 myserver systemd[1]: var-lib-docker-overlay2-9e89cfd7fe2534b920d678e541930ad0243337915ad18e272a48270054a3de29-merged.mount: Deactivated successfully.
...
Feb  4 01:19:18 myserver systemd[1]: run-docker-netns-001e2f44e8d5.mount: Deactivated successfully.
Feb  4 01:19:18 myserver dockerd[347660]: time="2024-02-04T01:19:18.659468754+01:00" level=info msg="Removing stale sandbox 001e2f44e8d5fd7ae7b6170e963ecdc58c5359789d6d191c249f92e251966888 (2325f38a1471dcb4b13d5fb47281fa48d642ab2883d57f8215eba83af4025f9d)"
Feb  4 01:19:18 myserver dockerd[347660]: time="2024-02-04T01:19:18.777075618+01:00" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint 44c340998ad181ca3a3d44c5c18ab4d8138db1ae5e5cb546a2e78138a2323d8d 289d638cbc715edc85b56887bce7a044bb2c52740e1e347247fb578196544058], retrying...."
Feb  4 01:19:18 myserver systemd[1]: run-docker-netns-001e2f44e8d5.mount: Deactivated successfully.
...
Feb  4 01:20:14 myserver dockerd[347660]: time="2024-02-04T01:20:14.713645699+01:00" level=error msg="2325f38a1471dcb4b13d5fb47281fa48d642ab2883d57f8215eba83af4025f9d cleanup: failed to delete container from containerd: container \"2325f38a1471dcb4b13d5fb47281fa48d642ab2883d57f8215eba83af4025f9d\" in namespace \"moby\": not found"
...
Feb  4 01:20:15 myserver systemd[1]: var-lib-docker-overlay2-2fa4d5c9490bb064222243fa793ca4b3a801bc578061fc56f94cd88ea77a8a03-merged.mount: Deactivated successfully.
Feb  4 01:20:15 myserver dockerd[347660]: time="2024-02-04T01:20:15.863382455+01:00" level=error msg="failed to start container" container=2325f38a1471dcb4b13d5fb47281fa48d642ab2883d57f8215eba83af4025f9d error="failed to create task for container: task 2325f38a1471dcb4b13d5fb47281fa48d642ab2883d57f8215eba83af4025f9d: already exists"
Feb  4 01:20:16 myserver systemd[1]: var-lib-docker-overlay2-8c5dc98fe07c5f22c34076339e9b3985dc60172a7cbecf23b60eb11b50d6349b-merged.mount: Deactivated successfully.
Feb  4 01:20:17 myserver dockerd[347660]: time="2024-02-04T01:20:17.120507735+01:00" level=warning msg="xtables contention detected while running [--wait -t nat -A DOCKER -p tcp -d 0/0 --dport 9001 -j DNAT --to-destination 192.168.0.2:9001 ! -i br-f2231fdee7e6]: Waited for 2.69 seconds and received \"\""
Feb  4 01:20:17 myserver systemd[1]: var-lib-docker-overlay2-4528cebfe1e019d91664e63bf6a24ea49280a771a52852565dea320971c3acfe-merged.mount: Deactivated successfully.
Feb  4 01:20:17 myserver dockerd[347660]: time="2024-02-04T01:20:17.620815879+01:00" level=error msg="failed to start container" container=9bdcd03834eb8a6c230016530da692d20f057da10475c936f040fb936639fae5 error="failed to create task for container: task 9bdcd03834eb8a6c230016530da692d20f057da10475c936f040fb936639fae5: already exists"
...