I'm running Docker on a bunch of hosts in AWS as part of an ECS cluster and over the last couple of weeks the Docker daemon has started to intermittently crash. I haven't changed anything at the system level (kernel, OS, Docker, etc.) and the overall workload is the same.
The system will run fine for days without any obvious pressure on CPU, memory, or IO. At some point the
dockerd-default process crashes and the load skyrockets to > 100 because (I believe) of the blocked processes. I've been unable to recover from this and my only option has been to terminate the instance.
Around the time that the daemon crashes
dmesg shows that the kernel is unable to schedule the process:
Apr 11 13:46:06 ip-172-31-84-92 kernel: [1356823.587223] INFO: task dockerd-default:1548 blocked for more than 120 seconds.
ps shows the process is in the
1055 ? Dsl 5684:45 /usr/bin/dockerd-default --bip=10.192.0.1/16 --fixed-cidr=10.192.0.0/16 --group=docker --pidfile=/var/run/docke.pid --storage-driver=overlay --raw-logs
lsof against the process to try to figure out what file handle could be holding everything up but nothing sticks out as obvious. Additionally, there isn't much disk IO so I don't think I'm being constrained from that perspective.
So, here are my questions:
- Has anyone encountered anything similar?
- Is there any way I can recover from this without terminating the instance?
- Any suggestions for next steps in debugging?
For reference, here are the versions of the various components: