2

I'm running Docker on a bunch of hosts in AWS as part of an ECS cluster and over the last couple of weeks the Docker daemon has started to intermittently crash. I haven't changed anything at the system level (kernel, OS, Docker, etc.) and the overall workload is the same.

The system will run fine for days without any obvious pressure on CPU, memory, or IO. At some point the dockerd-default process crashes and the load skyrockets to > 100 because (I believe) of the blocked processes. I've been unable to recover from this and my only option has been to terminate the instance.

Around the time that the daemon crashes dmesg shows that the kernel is unable to schedule the process:

Apr 11 13:46:06 ip-172-31-84-92 kernel: [1356823.587223] INFO: task dockerd-default:1548 blocked for more than 120 seconds.

ps shows the process is in the Dsl state:

1055 ?  Dsl  5684:45 /usr/bin/dockerd-default --bip=10.192.0.1/16 --fixed-cidr=10.192.0.0/16 --group=docker --pidfile=/var/run/docke.pid --storage-driver=overlay --raw-logs

I've run lsof against the process to try to figure out what file handle could be holding everything up but nothing sticks out as obvious. Additionally, there isn't much disk IO so I don't think I'm being constrained from that perspective.

So, here are my questions:

  • Has anyone encountered anything similar?
  • Is there any way I can recover from this without terminating the instance?
  • Any suggestions for next steps in debugging?

For reference, here are the versions of the various components:

Ubuntu: 14.04.5 LTS
Kernel: 4.14.12-041412-generic
Docker: 18.06.1-ce

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.