Issue Running Docker with 17.8.11 (Ubuntu 16.04)

[Dukemaster]

Hi,
since I changed my server I have a problem running docker. In Plesk control panel I get the message that there is no local docker node. The Plesk support link is empty.
I searched a lot and want to show you all of my output in the last box.
On my old server docker was running from the beginning. The new one has hardware raid, don't know if this is the reason, why docker isn't running.
Could You help me to solve this problem, please?

Is it true to simply have to run:

unmount /var/lib/docker/tmp/docker-aufs-union784040076

The other warnings I also don't know how to solve:

WARN[0001] Your kernel does not support swap memory limit
WARN[0001] Your kernel does not support cgroup rt period
WARN[0001] Your kernel does not support cgroup rt runtime

root@server:~# systemctl status docker.service
● docker.service - Docker Application Container Engine
   Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
   Active: active (running) since Sat 2018-04-21 14:26:43 CEST; 2 days ago
     Docs: https://docs.docker.com
Main PID: 2491 (dockerd)
   CGroup: /system.slice/docker.service
           ├─2491 /usr/bin/dockerd -H fd://
           └─3484 docker-containerd -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --metrics-interval=0 --start-timeout 2m --state-dir /var/run/docker/l

Apr 21 14:26:42 server.example.com dockerd[2491]: time="2018-04-21T14:26:42.454274675+02:00" level=warning msg="Your kernel does not support cgroup rt period"
Apr 21 14:26:42 server.example.com dockerd[2491]: time="2018-04-21T14:26:42.454280667+02:00" level=warning msg="Your kernel does not support cgroup rt runtime"
Apr 21 14:26:42 server.example.com dockerd[2491]: time="2018-04-21T14:26:42.454496900+02:00" level=info msg="Loading containers: start."
Apr 21 14:26:42 server.example.com dockerd[2491]: time="2018-04-21T14:26:42.693920078+02:00" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.
Apr 21 14:26:42 server.example.com dockerd[2491]: time="2018-04-21T14:26:42.711472547+02:00" level=info msg="Loading containers: done."
Apr 21 14:26:42 server.example.com dockerd[2491]: time="2018-04-21T14:26:42.722403018+02:00" level=warning msg="Couldn't run auplink before unmount /var/lib/docker/tmp/dock
Apr 21 14:26:43 server.example.com dockerd[2491]: time="2018-04-21T14:26:43.261676135+02:00" level=info msg="Daemon has completed initialization"
Apr 21 14:26:43 server.example.com dockerd[2491]: time="2018-04-21T14:26:43.261696206+02:00" level=info msg="Docker daemon" commit=89658be graphdriver=aufs version=17.05.0-
Apr 21 14:26:43 server.example.com dockerd[2491]: time="2018-04-21T14:26:43.265587092+02:00" level=info msg="API listen on /var/run/docker.sock"
Apr 21 14:26:43 server.example.com systemd[1]: Started Docker Application Container Engine.
root@server:~# ^C
root@server:~# ll /usr/bin | grep docker
-rwxr-xr-x  1 root   root    18454184 May  5  2017 docker*
-rwxr-xr-x  1 root   root    11422736 May  5  2017 docker-containerd*
-rwxr-xr-x  1 root   root    10575776 May  5  2017 docker-containerd-ctr*
-rwxr-xr-x  1 root   root     1976648 May  5  2017 docker-containerd-shim*
-rwxr-xr-x  1 root   root    38005136 May  5  2017 dockerd*
-rwxr-xr-x  1 root   root      862296 May  5  2017 docker-init*
-rwxr-xr-x  1 root   root     2573800 May  5  2017 docker-proxy*
-rwxr-xr-x  1 root   root     8199696 May  5  2017 docker-runc*
root@server:~# docker --version
Docker version 17.05.0-ce, build 89658be
root@server:~# dockerd --version
Docker version 17.05.0-ce, build 89658be
root@server:~# docker-containerd --version
containerd version 0.2.3 commit: 9048e5e50717ea4497b757314bad98ea3763c145
root@server:~# docker-containerd-ctr --version
ctr version 0.2.3 commit: 9048e5e50717ea4497b757314bad98ea3763c145
root@server:~# docker-init --version
tini version 0.13.0 - git.949e6fa
root@server:~# docker-runc --version
runc version 1.0.0-rc2
commit: 9c2d8d184e5da67c95d601382adf14862e4f2228
spec: 1.0.0-rc2-dev
root@server:~# uname -a
Linux server.arox.eu 4.4.0-119-generic #143-Ubuntu SMP Mon Apr 2 16:08:24 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
root@server:~# file /usr/bin/docker-runc
/usr/bin/docker-runc: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=2618bd80978a596bc84c4dc8cf81be9cebef43ce, not stripped
root@server:~# docker daemon -D
Command "daemon" is deprecated, and will be removed in Docker 17.12. Please run `dockerd` directly.
Error starting daemon: pid file found, ensure docker is not running or delete /var/run/docker.pid
root@server:~# dockerd
Error starting daemon: pid file found, ensure docker is not running or delete /var/run/docker.pid
root@server:~# service docker stop
root@server:~# dockerd
INFO[0000] libcontainerd: new containerd process, pid: 30209
WARN[0000] containerd: low RLIMIT_NOFILE changing to max  current=1024 max=1048576
WARN[0001] failed to rename /var/lib/docker/tmp for background deletion: %!s(<nil>). Deleting synchronously
INFO[0001] [graphdriver] using prior storage driver: aufs
INFO[0001] Graph migration to content-addressability took 0.00 seconds
WARN[0001] Your kernel does not support swap memory limit
WARN[0001] Your kernel does not support cgroup rt period
WARN[0001] Your kernel does not support cgroup rt runtime
INFO[0001] Loading containers: start.
INFO[0001] Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address
INFO[0001] Loading containers: done.
WARN[0001] Couldn't run auplink before unmount /var/lib/docker/tmp/docker-aufs-union784040076: exec: "auplink": executable file not found in $PATH
INFO[0001] Daemon has completed initialization
INFO[0001] Docker daemon                                 commit=89658be graphdriver=aufs version=17.05.0-ce
INFO[0001] API listen on /var/run/docker.sock
^CINFO[0151] Processing signal 'interrupt'
INFO[0151] stopping containerd after receiving terminated
root@server:~# ^C
root@server:~#

 

[Viktor Erpylev]

Hi.
First, for the warnings about cgroups look for this article https://support.plesk.com/hc/en-us/articles/115004786934-Docker-service-shows-warnings-Your-kernel-does-not-support-cgroup-rt-runtime


After kernel updates, docker should work. If not, you have troubles with AUFS file system. According to this warning

Couldn't run auplink before unmount /var/lib/docker/tmp/docker-aufs-union784040076: exec: "auplink": executable file not found in $PATH

auplink is a part of aufs-tools package, so please check installation apt install -y aufs-tools.
Also, you should check prerequisites part of this article https://docs.docker.com/storage/storagedriver/aufs-driver/
The main check here: aufs kernel support

$ grep aufs /proc/filesystems
nodev   aufs

It's better to use overlay2. You can easily configure docker daemon - just create or modify /etc/docker/daemon.json file, add the following contents:

{
  "storage-driver": "overlay2"
}

Note: if you do it on a working system you can lose your data. You can find more information here https://docs.docker.com/storage/storagedriver/overlayfs-driver/

 

[Dukemaster]

Hi @Viktor Erpylev Thank You very much for your detailed instructions.
By the first check I get the same expected result.

root@server:~# grep aufs /proc/filesystems
nodev   aufs

Then I created the /etc/docker/daemon.json file and switched from aufs to overlay2
Now the docker info is:

root@server:~# docker info
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 17.05.0-ce
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9048e5e50717ea4497b757314bad98ea3763c145
runc version: 9c2d8d184e5da67c95d601382adf14862e4f2228
init version: 949e6fa
Security Options:
apparmor
seccomp
  Profile: default
Kernel Version: 4.4.0-121-generic
Operating System: Ubuntu 16.04.4 LTS
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 31.26GiB
Name: server.example.com
ID: TGL2:7UWR:KEG6:B32W:A37H:NYV7:HBMT:S42K:UOSW:BCF7:TBLO:CKCM
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

Could you please help me a little further?

 

[Viktor Erpylev]

Great, we have last warning. Docker documentation provides us a workaround to enable swap limit: Post-installation steps for Linux

1. Log into the Ubuntu or Debian host as a user with sudo privileges.
   
   
2. Edit the /etc/default/grub file. Add or edit the GRUB_CMDLINE_LINUX line to add the following two key-value pairs:

   GRUB_CMDLINE_LINUX="cgroup_enable=memory swapaccount=1"

   
   
3. Update GRUB.
   

   $ sudo update-grub

The changes take effect when the system is rebooted.

 

[Dukemaster]

I did this advice. After update grub the server get lost. Now it is not reachable over shell. I never did something with grub before. Changed to this newer 1and1 server with hardware raid 4 weeks ago.
Serial console offers me a lot of options. I don't know what do to do now.
I don't need docker. Your advice looked so easy, so simple. I was happy about. Now server is unreachable by normal tty, only over serial console. But this output in serial console is not usual (for me).
I rebootet over 1and1 control panel rescue by normal reboot, also normal reboot for next reboot option. The screenshot shows the dozen of options running during the rescue reboot.
PLEASE, could you @Viktor Erpylev help me what to do. I'm trusting you. Perhaps I should have done something special by setting up the server 4 weeks ago, I don't know.
This is only a little problem, I feel it. But I don't know how to come out of this trap.
Greets

 

[Viktor Erpylev]

Hi, Dukemaster.
Sorry that you in such situation because of me.
Serial console - is the last chance to rescue your server - it's the hardest way.
I suggest using KVM (Keyboard, mouse, and monitor attached to your server and redirected via network). Accessing Server via the KVM console
You would see your server like you directly connect to it.

Your plan look's like this:

1. Login into recovery mode
2. Revert changes in /etc/default/grub
3. Run update-grub
4. Reboot

Some screenshots here How to Fix an Ubuntu System When It Won’t Boot
You need "Use Recovery Mode If You Can Access GRUB" part of How-to.
Choose "root Drop to root shell prompt" to get a shell. (grub option wouldn't help because it rewrites wrong boot settings to disk again)
"mount -o remount,rw /" command would help you to change file system state from read-only to writable.


This is a little problem, yes, but if you have any doubts or you feel not enough Force inside - try to find somebody who can do it instead of you or make the full backup of your server.

 

[Brujo]

1&1 has a good documentation for the rescue like:
1&1 Help Center
Mount a Partition in the Linux Rescue System - 1&1 Help Center
Mit dem Linux-Rescue-System arbeiten - 1&1 Hilfe Center

which should help you to revert the change especial if you have to mount the volumes chroot and so on...

 

[Dukemaster]

Hi @Viktor Erpylev, no problem. First, you are not responsible for my less knowledge in this case. 2nd you didn't leave me alone with my problem. 3rd you are sympathetic and helpful (like all of Plesk staff members I read here and support). THANKS a lot.
I'm close to solve it now and only have one little problem. I was able to delete the line

GRUB_CMDLINE_LINUX="cgroup_enable=memory swapaccount=1"

in grub by:

mount /dev/sda1 /mnt

The big problem now is that I can't execute command "update-grub" in this stage.
I have a support article by 1and1 (sorry no direct english version for this) but it seems a little older article, most problem is the /opt/ device in my environment. Because we shouldn't forget the software/hardware raid differences.
Here is my fstab with mounted sda1:

rescue:~# mount /dev/sda1 /mnt
rescue:~# cat /mnt/etc/fstab
/dev/sda1       /               ext3    defaults,noatime,barrier=0       1 1
/dev/sda2       none            swap    sw
/dev/vg00/usr   /usr            ext4    noatime,errors=remount-ro       0 2
/dev/vg00/var   /var            ext4    noatime,errors=remount-ro       0 2
/dev/vg00/home  /home           ext4    noatime,errors=remount-ro       0 2
/dev/vg00/opt   /opt            ext4    noatime,errors=remount-ro       0 2
#hdddata /dev/hdd/data  /data           ext4    noatime,errors=remount-ro       0 2
proc            /proc           proc    nodev,noexec,nosuid     0 0

According to the article and the advice (I'm not sure if this is necessary at all) to also mount the others, switch into chroot to be able to update grub.

rescue:~# mount /dev/vg00/usr /mnt/usr
rescue:~# mount /dev/vg00/var /mnt/var
rescue:~# mount /dev/vg00/home /mnt/home
rescue:~# for f in proc sys dev ; do mount --bind /$f /mnt/$f ; done
# Depends the file systems /proc, /sys and /dev with the single command under /mnt

and

rescue:~# chroot /mnt

.
After updating grub file unmounting all devices/partitions and reboot.

Do I really need this only for upgrading grub or is there another way to execute this little command?

THANKS @Brujo too...sorry for my late statement, it took me over 30 minutes to write this answer. I didn't realize your posting. But now I take a closer look about your help.
Greets

 

[Dukemaster]

THANKS A LOT MY PLESK FRIENDS !
Server is back again. It was hard in the end, I had to merge the english and the german support articles, each one has little, but important differences.
Sorry to take your time, especially for the reason that it was not Plesk related. Wish I would know what happened wrong.
@Viktor Erpylev please don't stop giving real good advices. Sometimes such things happen because providers have their own special and additional way to secure their systems, also differences in OS's and modern web technolgies. Plesk has grown so amazing in this fast Internet world, so problems are normal....

Have a nice weekend.

 

[Dukemaster]

End good - everything good...
Thanks and sorry again @Viktor Erpylev
I thought over and over what the mistake was, my mistake. I read the docker docs you gave me here in your post.
On the day I made it, I was not wake at work, tired...
My mistake was to write the two definitions not in the already existing line , I make a new (duplicate) one.
WRONG:

GRUB_CMDLINE_LINUX="console=ttyS0,57600 console=tty0 net.ifnames=0 biosdevname=0"
GRUB_CMDLINE_LINUX="cgroup_enable=memory swapaccount=1"

That was the reason why grub was updated without errors/warnings, but after reboot linux was not able to handle it.
Few minutes ago I did it again, this time I added the two commands in the ALREADY EXISTING line, updated grub..reboot...without errors.
CORRECT:

GRUB_CMDLINE_LINUX="console=ttyS0,57600 console=tty0 net.ifnames=0 biosdevname=0 cgroup_enable=memory swapaccount=1"

Now, Docker (Plesk Panel) is already the information about using only local images or buy it.

Perhaps another configuration is missing. Do you have any idea?
Docker is running all the time...

root@server:~# systemctl status docker.service
● docker.service - Docker Application Container Engine
   Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2018-05-02 22:21:03 CEST; 1h 10min ago
     Docs: https://docs.docker.com
 Main PID: 2522 (dockerd)
    Tasks: 30
   Memory: 55.5M
      CPU: 8.977s
   CGroup: /system.slice/docker.service
           ├─2522 /usr/bin/dockerd -H fd://
           └─3524 docker-containerd -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --metrics-interval=0 --start-timeout 2m --state-dir /var/run/docker/l

May 02 22:21:02 server.arox.eu dockerd[2522]: time="2018-05-02T22:21:02.205920801+02:00" level=info msg="Graph migration to content-addressability took 0.00 seconds"
May 02 22:21:02 server.arox.eu dockerd[2522]: time="2018-05-02T22:21:02.206214102+02:00" level=warning msg="Your kernel does not support cgroup rt period"
May 02 22:21:02 server.arox.eu dockerd[2522]: time="2018-05-02T22:21:02.206228281+02:00" level=warning msg="Your kernel does not support cgroup rt runtime"
May 02 22:21:02 server.arox.eu dockerd[2522]: time="2018-05-02T22:21:02.206454533+02:00" level=info msg="Loading containers: start."
May 02 22:21:02 server.arox.eu dockerd[2522]: time="2018-05-02T22:21:02.557261134+02:00" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.
May 02 22:21:02 server.arox.eu dockerd[2522]: time="2018-05-02T22:21:02.573191022+02:00" level=info msg="Loading containers: done."
May 02 22:21:03 server.arox.eu dockerd[2522]: time="2018-05-02T22:21:03.843578782+02:00" level=info msg="Daemon has completed initialization"
May 02 22:21:03 server.arox.eu dockerd[2522]: time="2018-05-02T22:21:03.843634600+02:00" level=info msg="Docker daemon" commit=89658be graphdriver=overlay2 version=17.0
May 02 22:21:03 server.arox.eu dockerd[2522]: time="2018-05-02T22:21:03.849760804+02:00" level=info msg="API listen on /var/run/docker.sock"
May 02 22:21:03 server.arox.eu systemd[1]: Started Docker Application Container Engine.
root@server:~#

 

[Ruslan Kosolapov]

Hi @Dukemaster

The problem itself is unclear for me, I mean I can't catch what's wrong
Could you provide the exact error message, or a screenshot of the Plesk page?