13

I have successfully upgraded my Debian buster to latest version (Bullseye) and after that, whenever I want to reboot or shut it down, it takes several minutes to complete while waiting for some process to finish with these messages:

watchdog: watchdog0: watchdog did not stop!
systemd-shutdown[1]: Syncing filesystem and block devices.
systemd-shutdown[1]: Sending SIGTERM to remaining process...
systemd-journald[372]: Received SIGTERM from PID 1 (systemd-shutdown).
systemd-shutdown[1]: waiting for process: containerd-shim.

I've got docker installed on my system which seems to be the cause of problem.

$ ps aux | grep containerd-shim
root        3420  0.0  0.1 1451744 21876 ?       Sl   11:07   0:00 /usr/bin/containerd-shim-runc-v2 -namespace moby -id 0dd6b89a62d...66cc5c0a44b6f01d77c -address /run/containerd/containerd.sock

$ dpkg -S /usr/bin/containerd-shim-runc-v2 
containerd: /usr/bin/containerd-shim-runc-v2

$ aptitude why containerd
i   docker.io Depends containerd

I've tried stopping dockers' service/socket before rebooting the system. Nothing changed.

Any idea how to resolve this?

$ docker version
Client:
 Version:           20.10.5+dfsg1
 API version:       1.41
 Go version:        go1.15.9
 Git commit:        55c4c88
 Built:             Wed Aug  4 19:55:57 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          20.10.5+dfsg1
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.15.9
  Git commit:       363e9a8
  Built:            Wed Aug  4 19:55:57 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.5~ds1
  GitCommit:        1.4.5~ds1-2
 runc:
  Version:          1.0.0~rc93+ds1
  GitCommit:        1.0.0~rc93+ds1-5+b2
 docker-init:
  Version:          0.19.0
  GitCommit:
Improve this question
1
  • dockerd uses containerd for creating containers but it is a separate process. It seems your containerd is the one who is taking time to close. Maybe it could be docker but the dmesg logs + containerd logs should tell you what was the issue if you keep your logs for couple of days.
    – arif
    Sep 1, 2021 at 21:46

1 Answer 1

Reset to default
9

If you're using live-restore, then this is a known issue of containerd shim v2. Fixed in containerd ≥ 1.6.0-rc.2. Backported to containerd 1.5.10. Fixed in Docker Engine 20.10.13.

Workaround:

  1. Use shim v1 instead of v2

    Either specify runtime for specific container

    docker run --runtime=io.containerd.runtime.v1.linux ...`

    Or set a system-wide default runtime

    # cat /etc/docker/daemon.json
{
    "default-runtime": "io.containerd.runtime.v1.linux",
    "live-restore": true
}

    WARNING: shim v1 is deprecated and will be removed in the next Docker release. WARNING: shim v1 doesn't work with cgroups v2

  2. Use a systemd service to kill containers before reboot/shutdown

    # /etc/systemd/system/containerd-shim-v2-workaround.service
[Unit]
Description=containerd-shim v2 workaround
Before=docker.service
Requires=containerd.service
After=containerd.service

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStop=-/bin/sh -c '[ "$(systemctl is-system-running)" = "stopping" ] || exit 0; ctr -n moby tasks ls -q | xargs -r -L1 ctr -n moby tasks kill; ctr -n moby containers ls -q | xargs -r ctr -n moby containers rm'

[Install]
WantedBy=containerd.service

    I posted this under the moby issue thread. But this may not suit every use case. Use with caution.

Improve this answer
2
  • 1
    Wanted to use shim v1 as a workaround since I experience the same issue but shim v1 doesn't work with cgroups v2 and fails during startup.
    – cmur2
    Mar 8, 2022 at 11:27
  • 1
    @cmur2 I edited my answer and add your comment to the warning.
    – youfu
    Mar 9, 2022 at 8:25

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .