Docker filesystem has no open files but cannot unmount

Question

I have an ext4 filesystem mounted on /var/lib/docker, using Ubuntu 14.04.

I ran umount /var/lib/docker and got:

umount: /var/lib/docker: device is busy.

I had already shut down docker with service docker stop and ps aux showed no processes.

The strange thing is that both fuser -M /var/lib/docker and lsof +f -- /var/lib/docker show no processes with files open.

This doesn't happen on 40+ other servers with same setup, and I have just noticed this.

A reboot fixed this - curious if anyone has some idea what's going on, or any other ways to investigate if it recurs.

• Docker (1.11.2) had been using aufs as the overlay filesystem for its containers, on top of this FS - but all containers and Docker processes were killed before the umount.
• Kernel details: Linux node05 4.4.0-38-generic #57~14.04.1-Ubuntu SMP Tue Sep 6 17:20:43 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

Possibly related questions:

• Can't unmount a loop filesystem but no open files
• What's preventing unmount of a bind mount?

UPDATE - recurrence on another server

This recurred on a different server with very similar configuration, in same situation - have shut down docker, and can't umount /var/lib/docker because it's "busy".

As before, fuser and lsof show no processes with files open on the docker FS.

mount | grep aufs showed no aufs filesystems

df -h| grep docker showed two Docker managed filesystems mounted underneath the /var/lib/docker directory (lsof doesn't show these, it seems).

However, unmounting these two filesystems didn't help with this problem.

Workaround

The only workaround remains to edit /etc/fstab to temporarily comment out the /var/lib/docker filesystem, then reboot the server so /var/lib/docker is not mounted.