Issue Docker can't start anymore

[keuta]

Server operating system version

CentOS Linux 7.9.2009 (Core)

Plesk version and microupdate number

Plesk Obsidian Web Pro Edition Version 18.0.49 Mise à jour n° 2

Hello there !
since the last night update (2023/02/03), the docker service won't start.
The update that broke it was :
" yum history info 1653
Modules complémentaires chargés : fastestmirror
ID de transaction : 1653
Temps de début : Fri Feb 3 03:29:39 2023
Début de RPMDB : 1333:d3417d62f1ed0b99c05361a94aa253c0b0c22676
Temps de fin : 03:30:05 2023 (26 secondes)
Fin de RPMDB : 1335:14f69a06e596ddd96c72c79abfce817fc60bb686
Utilisateur : Système <indéfini>
Code retour : Réussi
Transaction effectuée avec :
Installé rpm-4.11.3-48.el7_9.x86_64 @updates
Installé yum-3.4.3-168.el7.centos.noarch @base
Installé yum-metadata-parser-1.1.4-10.el7.x86_64 @anaconda
Installé yum-plugin-fastestmirror-1.1.31-54.el7_8.noarch @updates
Paquets modifiés :
Installation déps. docker-buildx-plugin-0.10.2-1.el7.x86_64 @plesk-ext-docker
Mis à jour docker-ce-3:20.10.23-3.el7.x86_64 @plesk-ext-docker
Mise à jour 3:23.0.0-1.el7.x86_64 @plesk-ext-docker
Mis à jour docker-ce-cli-1:20.10.23-3.el7.x86_64 @plesk-ext-docker
Mise à jour 1:23.0.0-1.el7.x86_64 @plesk-ext-docker
Mis à jour docker-ce-rootless-extras-20.10.23-3.el7.x86_64 @plesk-ext-docker
Mise à jour 23.0.0-1.el7.x86_64 @plesk-ext-docker
Installation déps. docker-compose-plugin-2.15.1-3.el7.x86_64 @plesk-ext-docker"

Must I just rollback or is there any other way to solve it ?

 

[Peter Debik]

Besides the yum output, what happens when you start Docker? What error messages can you see or do you find any in log files?

 

[keuta]

Hi Peter !
Here's the message when i try to launch the docker service :

Unable to start the service : Job for docker.service failed because the control process exited with error code. See "systemctl status docker.service" and "journalctl -xe" for details.

Here's the result of "systemctl status docker.service" :

docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
Active: failed (Result: start-limit) since ven. 2023-02-03 11:50:57 CET; 1min 38s ago
Docs: Docker Documentation
Process: 6476 ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock (code=exited, status=1/FAILURE)
Main PID: 6476 (code=exited, status=1/FAILURE)

févr. 03 11:50:55 ns3012771.ip-149-202-74.eu systemd[1]: Failed to start Docker Application Container Engine.
févr. 03 11:50:55 ns3012771.ip-149-202-74.eu systemd[1]: Unit docker.service entered failed state.
févr. 03 11:50:55 ns3012771.ip-149-202-74.eu systemd[1]: docker.service failed.
févr. 03 11:50:57 ns3012771.ip-149-202-74.eu systemd[1]: docker.service holdoff time over, scheduling restart.
févr. 03 11:50:57 ns3012771.ip-149-202-74.eu systemd[1]: Stopped Docker Application Container Engine.
févr. 03 11:50:57 ns3012771.ip-149-202-74.eu systemd[1]: start request repeated too quickly for docker.service
févr. 03 11:50:57 ns3012771.ip-149-202-74.eu systemd[1]: Failed to start Docker Application Container Engine.
févr. 03 11:50:57 ns3012771.ip-149-202-74.eu systemd[1]: Unit docker.service entered failed state.
févr. 03 11:50:57 ns3012771.ip-149-202-74.eu systemd[1]: docker.service failed.

and "journalctl -xe" :

févr. 03 11:54:00 ns3012771.ip-149-202-74.eu dockerd[8320]: time="2023-02-03T11:54:00.828284053+01:00" level=info msg="[core] [Channel #1] parsed dial target is: {Scheme:unix Authority: Endpoint:run/containerd/containerd.sock URL:{Scheme:u
févr. 03 11:54:00 ns3012771.ip-149-202-74.eu dockerd[8320]: time="2023-02-03T11:54:00.828293368+01:00" level=info msg="[core] [Channel #1] Channel authority set to \"localhost\"" module=grpc
févr. 03 11:54:00 ns3012771.ip-149-202-74.eu dockerd[8320]: time="2023-02-03T11:54:00.828389753+01:00" level=info msg="[core] [Channel #1] Resolver state updated: {\n \"Addresses\": [\n {\n \"Addr\": \"/run/containerd/containerd.s
févr. 03 11:54:00 ns3012771.ip-149-202-74.eu dockerd[8320]: time="2023-02-03T11:54:00.828447076+01:00" level=info msg="[core] [Channel #1] Channel switches to new LB policy \"pick_first\"" module=grpc
févr. 03 11:54:00 ns3012771.ip-149-202-74.eu dockerd[8320]: time="2023-02-03T11:54:00.828508435+01:00" level=info msg="[core] [Channel #1 SubChannel #2] Subchannel created" module=grpc
févr. 03 11:54:00 ns3012771.ip-149-202-74.eu dockerd[8320]: time="2023-02-03T11:54:00.828552001+01:00" level=info msg="[core] [Channel #1 SubChannel #2] Subchannel Connectivity change to CONNECTING" module=grpc
févr. 03 11:54:00 ns3012771.ip-149-202-74.eu dockerd[8320]: time="2023-02-03T11:54:00.828580536+01:00" level=info msg="[core] [Channel #1 SubChannel #2] Subchannel picks a new address \"/run/containerd/containerd.sock\" to connect" module=
févr. 03 11:54:00 ns3012771.ip-149-202-74.eu dockerd[8320]: time="2023-02-03T11:54:00.828582380+01:00" level=info msg="[core] [Channel #1] Channel Connectivity change to CONNECTING" module=grpc
févr. 03 11:54:00 ns3012771.ip-149-202-74.eu dockerd[8320]: time="2023-02-03T11:54:00.828759931+01:00" level=info msg="[core] [Channel #1 SubChannel #2] Subchannel Connectivity change to READY" module=grpc
févr. 03 11:54:00 ns3012771.ip-149-202-74.eu dockerd[8320]: time="2023-02-03T11:54:00.828779152+01:00" level=info msg="[core] [Channel #1] Channel Connectivity change to READY" module=grpc
févr. 03 11:54:00 ns3012771.ip-149-202-74.eu dockerd[8320]: time="2023-02-03T11:54:00.829279125+01:00" level=info msg="[core] [Channel #4] Channel created" module=grpc
févr. 03 11:54:00 ns3012771.ip-149-202-74.eu dockerd[8320]: time="2023-02-03T11:54:00.829291271+01:00" level=info msg="[core] [Channel #4] original dial target is: \"unix:///run/containerd/containerd.sock\"" module=grpc
févr. 03 11:54:00 ns3012771.ip-149-202-74.eu dockerd[8320]: time="2023-02-03T11:54:00.829307163+01:00" level=info msg="[core] [Channel #4] parsed dial target is: {Scheme:unix Authority: Endpoint:run/containerd/containerd.sock URL:{Scheme:u
févr. 03 11:54:00 ns3012771.ip-149-202-74.eu dockerd[8320]: time="2023-02-03T11:54:00.829316176+01:00" level=info msg="[core] [Channel #4] Channel authority set to \"localhost\"" module=grpc
févr. 03 11:54:00 ns3012771.ip-149-202-74.eu dockerd[8320]: time="2023-02-03T11:54:00.829336117+01:00" level=info msg="[core] [Channel #4] Resolver state updated: {\n \"Addresses\": [\n {\n \"Addr\": \"/run/containerd/containerd.s
févr. 03 11:54:00 ns3012771.ip-149-202-74.eu dockerd[8320]: time="2023-02-03T11:54:00.829349175+01:00" level=info msg="[core] [Channel #4] Channel switches to new LB policy \"pick_first\"" module=grpc
févr. 03 11:54:00 ns3012771.ip-149-202-74.eu dockerd[8320]: time="2023-02-03T11:54:00.829358643+01:00" level=info msg="[core] [Channel #4 SubChannel #5] Subchannel created" module=grpc
févr. 03 11:54:00 ns3012771.ip-149-202-74.eu dockerd[8320]: time="2023-02-03T11:54:00.829379033+01:00" level=info msg="[core] [Channel #4 SubChannel #5] Subchannel Connectivity change to CONNECTING" module=grpc
févr. 03 11:54:00 ns3012771.ip-149-202-74.eu dockerd[8320]: time="2023-02-03T11:54:00.829408364+01:00" level=info msg="[core] [Channel #4 SubChannel #5] Subchannel picks a new address \"/run/containerd/containerd.sock\" to connect" module=
févr. 03 11:54:00 ns3012771.ip-149-202-74.eu dockerd[8320]: time="2023-02-03T11:54:00.829413933+01:00" level=info msg="[core] [Channel #4] Channel Connectivity change to CONNECTING" module=grpc
févr. 03 11:54:00 ns3012771.ip-149-202-74.eu dockerd[8320]: time="2023-02-03T11:54:00.829521580+01:00" level=info msg="[core] [Channel #4 SubChannel #5] Subchannel Connectivity change to READY" module=grpc
févr. 03 11:54:00 ns3012771.ip-149-202-74.eu dockerd[8320]: time="2023-02-03T11:54:00.829537291+01:00" level=info msg="[core] [Channel #4] Channel Connectivity change to READY" module=grpc
févr. 03 11:54:00 ns3012771.ip-149-202-74.eu dockerd[8320]: time="2023-02-03T11:54:00.935780537+01:00" level=warning msg="Usage of loopback devices is strongly discouraged for production use. Please use `--storage-opt dm.thinpooldev` or us
févr. 03 11:54:00 ns3012771.ip-149-202-74.eu dockerd[8320]: time="2023-02-03T11:54:00.941888979+01:00" level=warning msg="Base device already exists and has filesystem xfs on it. User specified filesystem will be ignored." storage-driver=
févr. 03 11:54:00 ns3012771.ip-149-202-74.eu dockerd[8320]: time="2023-02-03T11:54:00.943725377+01:00" level=error msg="[graphdriver] prior storage driver devicemapper is deprecated and will be removed in a future release; update the the d
févr. 03 11:54:00 ns3012771.ip-149-202-74.eu dockerd[8320]: time="2023-02-03T11:54:00.943867667+01:00" level=info msg="[core] [Channel #1] Channel Connectivity change to SHUTDOWN" module=grpc
févr. 03 11:54:00 ns3012771.ip-149-202-74.eu dockerd[8320]: time="2023-02-03T11:54:00.943892024+01:00" level=info msg="[core] [Channel #1 SubChannel #2] Subchannel Connectivity change to SHUTDOWN" module=grpc
févr. 03 11:54:00 ns3012771.ip-149-202-74.eu dockerd[8320]: time="2023-02-03T11:54:00.943903486+01:00" level=info msg="[core] [Channel #1 SubChannel #2] Subchannel deleted" module=grpc
févr. 03 11:54:00 ns3012771.ip-149-202-74.eu dockerd[8320]: time="2023-02-03T11:54:00.943908531+01:00" level=info msg="[core] [Channel #1] Channel deleted" module=grpc
févr. 03 11:54:00 ns3012771.ip-149-202-74.eu dockerd[8320]: failed to start daemon: error initializing graphdriver: prior storage driver devicemapper is deprecated and will be removed in a future release; update the the daemon configuratio
févr. 03 11:54:00 ns3012771.ip-149-202-74.eu systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE
févr. 03 11:54:00 ns3012771.ip-149-202-74.eu systemd[1]: Failed to start Docker Application Container Engine.
-- Subject: L'unité (unit) docker.service a échoué
-- Defined-By: systemd
-- Support: systemd-devel Info Page
--
-- L'unité (unit) docker.service a échoué, avec le résultat failed.
févr. 03 11:54:00 ns3012771.ip-149-202-74.eu systemd[1]: Unit docker.service entered failed state.
févr. 03 11:54:00 ns3012771.ip-149-202-74.eu systemd[1]: docker.service failed.
févr. 03 11:54:01 ns3012771.ip-149-202-74.eu systemd[1]: Created slice User Slice of root.
-- Subject: L'unité (unit) user-0.slice a terminé son démarrage
-- Defined-By: systemd
-- Support: systemd-devel Info Page

I think that this line is relevant :

Févr. 03 11:54:00 ns3012771.ip-149-202-74.eu dockerd[8320]: failed to start daemon: error initializing graphdriver: prior storage driver devicemapper is deprecated and will be removed in a future release; update the the daemon configuration

Thank you for your concern.

 

[Peter Debik]

The line is right, but the solution involves for sure some OS configuration tweaks. None of the articles I have on file address exactly this issue and "update the daemon configuration" can mean a lot. Your best option is to submit this to Plesk support, because an engineer will need to look into the server to fix it directly on your machine.

How to get support directly from Plesk?

Question How to get support directly from Plesk? Answer Lease-type Plesk licenses purchased directly from Plesk running on supported Plesk version include full free support. Perpetual-type license...

support.plesk.com

 

[MSWeb]

I have the exact same issue, Docker will not work on my server anymore. Also CentOS. Does Plesk offer support by now? Last time I was told to contact my license provider instead, but my license provider can not offer direct product support for Plesk for these specific Plesk-issues.

I need to get my server functional again as soon as possible, please try to get a help page created. Would it help to submit an issue to the Reports-section?

 

[MSWeb]

Sorry for the double-post, but I can not edit my previous answer anymore. Where can I find the configuration file for Docker, created by the Docker-plugin? I would definitely deploy those configruation tweaks myself, if it meant getting my services running sooner than later.
I have not changed the configuration up until now, I am solely relying on the Plesk Docker-plugin on this machine.

 

[Hempie]

I have this issue too, please repair this.

 

[keuta]

I rolledback to the previous docker's version and everything is fine now.

sudo yum history

will give you the last installations and updates, then do

sudo yum history info #numberofthelatestupgrade

to see if this was the update of docker, when you find the line that correspond with docker update just do

sudo yum history undo #numberofthedockerupgrade

and restart docker service

 

[Kaspar]

Last time I was told to contact my license provider instead, but my license provider can not offer direct product support for Plesk for these specific Plesk-issues.

Unfortunately I don't have a solution to the docker issues you're facing. However I do want point out that Plesk support is available for all Plesk users. However support might not be included in your license if you have obtained it trough a reseller (or was included with your VPS). In which case you can still purchase support from Plesk. First month is free. See: https://support.plesk.com/hc/en-us/articles/213953025-How-to-get-support-directly-from-Plesk-

Would it help to submit an issue to the Reports-section?

Yes, it would be helpful if you submit a detailed bug report about this issue. That way developers can investigate the issue. However if you need help on this urgently it's best to contact support.

EDIT: the solution posted from @keuta seems to remedy te issue.

 

[MSWeb]

The current version of Docker Engine 23.0 can also be used, when the storage driver is manually set:

daemon: require storage-driver to be set if the driver is deprecated by thaJeztah · Pull Request #43378 · moby/moby

relates to: #38017 #38018 #38090 #38019 daemon: graphdriver: some minor cleanup use pkg/errors for errors and fix error-capitalisation remove one redundant call to logDeprecatedWarning() (we're ...

github.com

But this will lead to a non-functional install when the storage driver is removed in the future. I would be great if Plesk Docker migrated the storage driver soon.

 

[MSWeb]

The daily auto-update re-installed the breaking change package. I have now disabled all auto-updates, except Plesk-core.

 

[GravuTrad]

Exactly same problem....

 

[GravuTrad]

The current version of Docker Engine 23.0 can also be used, when the storage driver is manually set:

daemon: require storage-driver to be set if the driver is deprecated by thaJeztah · Pull Request #43378 · moby/moby

relates to: #38017 #38018 #38090 #38019 daemon: graphdriver: some minor cleanup use pkg/errors for errors and fix error-capitalisation remove one redundant call to logDeprecatedWarning() (we're ...

github.com

But this will lead to a non-functional install when the storage driver is removed in the future. I would be great if Plesk Docker migrated the storage driver soon.

The problem seems clearly here.

 

[GravuTrad]

Use the OverlayFS storage driver

Learn how to optimize your use of OverlayFS driver.

docs.docker.com

Maybe?

 

[Peter Debik]

I'm afraid that this is an issue caused by a change in Docker since their version 23.0. Please see the sections Deprecated Engine Features and Deprecated Engine Features in the Docker documentation. Their documentation states: "Starting with the v23.0.0 release, the Docker Engine will fail to start if a deprecated storage driver is used."

We do not yet have a recommendation what the best solution for handling the issue is, because it is not caused by a Plesk fault, but rather a Docker issue, and the scenarios in customer installations have all kinds of settings and prerequisites that there is no clear "size fits all" solution path. I'll try to keep you posted here when more information becomes available. If you have individual solutions, please also keep posting.

 

[Peter Debik]

Before experimenting, please be sure to have a full backup of your Docker contents, because:

From the prerequisites on Docker docs:

Changing the storage driver makes existing containers and images inaccessible on the local system. Use docker save to save any images you have built or push them to Docker Hub or a private registry before changing the storage driver, so that you do not need to re-create them later.

So even if there are steps on how change storage driver to supported overlay2, the containers will be no longer available:
Use the OverlayFS storage driver

 

[GravuTrad]

I've done a ticket:
first answer:
The provided issue is looking similar to the issue with Docker that was found by our development team and currently is under investigation.

 

[MSWeb]

I'm afraid that this is an issue caused by a change in Docker since their version 23.0. Please see the sections Deprecated Engine Features and Deprecated Engine Features in the Docker documentation. Their documentation states: "Starting with the v23.0.0 release, the Docker Engine will fail to start if a deprecated storage driver is used."

We do not yet have a recommendation what the best solution for handling the issue is, because it is not caused by a Plesk fault, but rather a Docker issue, and the scenarios in customer installations have all kinds of settings and prerequisites that there is no clear "size fits all" solution path. I'll try to keep you posted here when more information becomes available. If you have individual solutions, please also keep posting.

Thank you for acknowledging the issue and providing more insight! In my case I rely on Plesk Docker for the configuration and have never manually touched the Docker engine-config. While I do not know if the extension actively set the storage driver in the past, more guidance on how to resolve the issue will be very welcome.

In my case recreating all the containers will not be fun, but feasible. For larger deployments it might be worse though, an automated solution would help a lot.

I've done a ticket:
first answer:
The provided issue is looking similar to the issue with Docker that was found by our development team and currently is under investigation.

Thank you! Great to know, that the team is actively working on it.

 

[GravuTrad]

Thedevicemapperstorage driver is deprecated in favor ofoverlay2, and will be removed in a future release. Users of thedevicemapperstorage driver are recommended to migrate to a different storage driver, such asoverlay2, which is now the default storage driver.

Device mapper storage driver

Deprecated in Release: v18.09 Disabled by default in Release: v23.0.0

a fix until an update of the plesk docker extension? (with overlay2?). Create this daemon file with this:

# cat /etc/docker/daemon.json
{
"storage-driver": "devicemapper"
}

 

[Peter Debik]

That update was suggested here internally, too, but after the change the docker service is up and running but Docker extension still shows the error "Local Docker node is missing" so I am not perfectly sure that this is a valid fix.