Busy Device on Umount

Question

I often experience a problem to umount a directory:

umount /mnt/dir
umount: /mnt/dir: device is busy

There are many reasons why the device is busy. Sometimes there are processes running which have open locks on it, sometimes there are other directories mounted on top of /mnt/dir.

My question:

What are the steps to check why a directory couldn't be unmounted.

I know there are many reasons, but it's ok if you explain a specific solution.

[EDIT]

[X] running processes on mounted volumes.
[X] another volume is mounted on top of a volume we want to unmount
[_] NFS locks the volume we want to unmount

Answer 1

The way to check is fuser -vm /mnt/dir, which must be run as root. It will tell you which processes are accessing the mount point.

An alternative is lsof /mnt/dir, which will show each open file on the mount. Again best run as root.

You can run either of these as non-root, but then the output will be limited to your processes—ones from other users will just be silently not shown, even though they will prevent unmounting the filesystem.

Example:

Watt:~# fuser -vm /mnt/Zia/src
                     USER        PID ACCESS COMMAND
/mnt/Zia/src:        root     kernel mount /mnt/Zia/src
                     anthony   24909 ..c.. bash
                     anthony   25041 F.c.. gvim

The "access" field tells you how its being accessed. In this case, the kernel has it in use as a mount (duh, but unmount will be OK with only this). bash has it as the current working directory (will have to cd to a different directory before unmount) and gvim both has the current directory and has a file open (will need to close that gvim).

Watt:~# lsof /mnt/Zia/src
COMMAND   PID    USER   FD   TYPE DEVICE SIZE/OFF    NODE NAME
bash    24909 anthony  cwd    DIR   0,26    12288 3527682 /mnt/Zia/src/perl (zia.vpn.home:/home/anthony/src)
gvim    25041 anthony  cwd    DIR   0,26    12288 3527682 /mnt/Zia/src/perl (zia.vpn.home:/home/anthony/src)
gvim    25041 anthony    6u   REG   0,26    16384 3526219 /mnt/Zia/src/perl/.utf8.c.swp (zia.vpn.home:/home/anthony/src)

In this output, you can see the current directories for both bash and gvim (as type DIR). You can also see which file gvim has open for write.

How to force the issue:

fuser has a -k option which will send a signal (default: SIGKILL) to each process using the mount. This is a rather forceful way to stop the mount from being busy. (And of course, be careful of what you SIGKILL!)

umount has an -l option to perform a lazy unmount. The mount will be removed from the filesystem namespace (so you won't see it under /mnt/Zia/src anymore, in the example) but it stays mounted, so programs accessing it can continue to do so. When the last program accessing it exits, the unmount will actually occur.

There is one final fixable cause of unmount failing, and that's an NFS server going down. Here you can use umount -f, but you risk data loss if you do so. (The client may have cached writes that haven't been confirmed by the server yet, and those writes will be discarded. Apps, however, have already been told the write is successful.)

Answer 2

You should use:

sudo umount -l <path>

From the man page:

-l, --lazy

Lazy unmount. Detach the filesystem from the file hierarchy now, and clean up all references to this filesystem as soon as it is not busy anymore.

A system reboot would be expected in near future if you're going to use this option for network filesystem or local filesystem with submounts.  The recommended use-case for umount -l is to prevent hangs on shutdown due to an unreachable network share where a normal umount will hang due to a downed server or a network partition. Remounts of the share will not be possible.

Answer 3

Another volume is mounted on top of a volume we want to unmount :

The mount command lets you know all the mounted volumes if invoqued without arguments nor options (except -v). You can have a list of active mountpoints by adding a bit of perl :

mount | perl -pe 's/.*on (\S+) type.*/\1/'

Then, just grep over the mointpoint from which you wish to unmount and you will know if there is mounted filesystems over this one.

mount | perl -pe 's/.*on (\S+) type.*/\1/' | grep '/mnt/dir/'

Then you have two solutions. Either unmount the filesystems, or move them with mount --move olddir newdir (kernel >2.5.1)

Answer 4

Open files

Processes with open files are the usual culprits. Display them:

lsof +f -- <mountpoint or device>

There is an advantage to using /dev/<device> rather than /mountpoint: a mountpoint will disappear after an umount -l, or it may be hidden by an overlaid mount.

fuser can also be used, but to my mind lsof has a more useful output. However fuser is useful when it comes to killing the processes causing your dramas so you can get on with your life.

List files on <mountpoint> (see caveat above):

fuser -vmM <mountpoint>

Interactively kill only processes with files open for writing:

fuser -vmMkiw <mountpoint>

After remounting read-only (mount -o remount,ro <mountpoint>), it is safe(r) to kill all remaining processes:

fuser -vmMk <mountpoint>

Mountpoints

The culprit can be the kernel itself. Another filesystem mounted on the filesystem you are trying to umount will cause grief. Check with:

mount | grep <mountpoint>/

For loopback mounts, also check the output of:

losetup -la

Anonymous inodes (Linux)

Anonymous inodes can be created by:

• Temporary files (open with O_TMPFILE)
• inotify watches
• [eventfd]
• [eventpoll]
• [timerfd]

These are the most elusive type of pokemon, and appear in lsof's TYPE column as a_inode (which is undocumented in the lsof man page).

They won't appear in lsof +f -- /dev/<device>, so you'll need to:

lsof | grep a_inode

For killing processes holding anonymous inodes, see: List current inotify watches (pathname, PID).

Answer 5

The question how to check if NFS access a directory which is going to unmount is still unanswered.

What I have is only this:

Check if nfsd is running:

pidof nfsd

Show mounted directories by clients:

showmount -a

and showmount w/o arguments shows only client hosts even if they are off line. I assume this is a special behavior of NFS.

Answer 6

For me, the issue was that I was logged in more than once (via ssh) and on one of the logins I was at command prompt where the pwd was inside a folder subordinate to the mount-point.

Answer 7

umount lazily and wait for mounts to be removed:

local mount
local attempt

umount -l "${mounts[@]}"

for mount in "${mounts[@]}"; do
    attempt=1

    while grep --quiet "${mount}" "/proc/mounts" && [[ "${attempt}" -le 50 ]]; do
        sleep 0.1
        attempt="$((attempt + 1))"
    done
done