39

I often experience a problem to umount a directory:

umount /mnt/dir
umount: /mnt/dir: device is busy

There are many reasons why the device is busy. Sometimes there are processes running which have open locks on it, sometimes there are other directories mounted on top of /mnt/dir.

My question:

What are the steps to check why a directory couldn't be unmounted.

I know there are many reasons, but it's ok if you explain a specific solution.

[EDIT]

[X] running processes on mounted volumes.
[X] another volume is mounted on top of a volume we want to unmount
[_] NFS locks the volume we want to unmount

72

The way to check is fuser -vm /mnt/dir, which must be run as root. It will tell you which processes are accessing the mount point.

An alternative is lsof /mnt/dir, which will show each open file on the mount. Again best run as root.

You can run either of these as non-root, but then the output will be limited to your processes—ones from other users will just be silently not shown, even though they will prevent unmounting the filesystem.

Example:

Watt:~# fuser -vm /mnt/Zia/src
                     USER        PID ACCESS COMMAND
/mnt/Zia/src:        root     kernel mount /mnt/Zia/src
                     anthony   24909 ..c.. bash
                     anthony   25041 F.c.. gvim

The "access" field tells you how its being accessed. In this case, the kernel has it in use as a mount (duh, but unmount will be OK with only this). bash has it as the current working directory (will have to cd to a different directory before unmount) and gvim both has the current directory and has a file open (will need to close that gvim).

Watt:~# lsof /mnt/Zia/src
COMMAND   PID    USER   FD   TYPE DEVICE SIZE/OFF    NODE NAME
bash    24909 anthony  cwd    DIR   0,26    12288 3527682 /mnt/Zia/src/perl (zia.vpn.home:/home/anthony/src)
gvim    25041 anthony  cwd    DIR   0,26    12288 3527682 /mnt/Zia/src/perl (zia.vpn.home:/home/anthony/src)
gvim    25041 anthony    6u   REG   0,26    16384 3526219 /mnt/Zia/src/perl/.utf8.c.swp (zia.vpn.home:/home/anthony/src)

In this output, you can see the current directories for both bash and gvim (as type DIR). You can also see which file gvim has open for write.

How to force the issue:

fuser has a -k option which will send a signal (default: SIGKILL) to each process using the mount. This is a rather forceful way to stop the mount from being busy. (And of course, be careful of what you SIGKILL!)

umount has an -l option to perform a lazy unmount. The mount will be removed from the filesystem namespace (so you won't see it under /mnt/Zia/src anymore, in the example) but it stays mounted, so programs accessing it can continue to do so. When the last program accessing it exits, the unmount will actually occur.

There is one final fixable cause of unmount failing, and that's an NFS server going down. Here you can use umount -f, but you risk data loss if you do so. (The client may have cached writes that haven't been confirmed by the server yet, and those writes will be discarded. Apps, however, have already been told the write is successful.)

  • 3
    Note that fuser -k is extremely risky, as you'll be doing it as root and if you're not very sure of which processes will be killed off you can do truly spectacular damage with a careless command... – Shadur Jan 4 '14 at 23:08
  • 1
    @Shadur well, hopefully you've already run it without the -k option, so you'll know which processes you're going to kill. But I'll add in a warning. – derobert Jan 5 '14 at 6:48
  • 1
    fuser -vm showed "kernel mount". had to do systemctl stop opt.mount instead of manual umount. – lkraav Aug 8 '15 at 18:36
  • 2
    For some reason umount -f doesn't work for me but running umount -l works perfectly. – Firze Oct 27 '15 at 9:44
  • Thanks for the note about umount -fand NFS. My issue was NFS related where my dev machines changed IPs and I could not get the share removed. – Eric Oct 4 '16 at 16:41
18

You should use:

sudo umount -l <path>
  • 6
    ⁺¹, I've no idea what silly person could downvote it. The -l is exactly the option to use when even -f doesn't work. – Hi-Angel Jan 11 '17 at 9:32
  • @Hi-Angel Because this is not what the OP is asking? – xhienne Jul 29 '17 at 11:23
  • @xheinne stack exchange is not just about answering a question like a stupeed bot, this answer is helpful. many people come from google search also. I personally found this helpful. The op shoudl accept the answer which he find relevant thats why the accept button is there. – user1735921 Sep 29 '17 at 23:43
6

Another volume is mounted on top of a volume we want to unmount :

The mount command lets you know all the mounted volumes if invoqued without arguments nor options (except -v). You can have a list of active mountpoints by adding a bit of perl :

mount | perl -pe 's/.*on (\S+) type.*/\1/'

Then, just grep over the mointpoint from which you wish to unmount and you will know if there is mounted filesystems over this one.

mount | perl -pe 's/.*on (\S+) type.*/\1/' | grep '/mnt/dir/'

Then you have two solutions. Either unmount the filesystems, or move them with mount --move olddir newdir (kernel >2.5.1)

  • 1
    Yes, thanks. /etc/mtab and /proc/mounts are also possible. – user55518 Jan 7 '14 at 10:32
  • Ah that's right, I always forgot those. Let's say that typing "mount" requires less characters (but more resources to execute?) – mveroone Jan 7 '14 at 14:28
  • 1
    I have an external USB storage device "permanently" mounted on a certain directory on my laptop, Sometimes the cable gets disconnected by mistake. It was a big pain to remount the device on the directory (because of "device busy") until I read this answer. Now I known to use mount --move olddir newdir. Thanks. – Silvio Levy Dec 30 '14 at 10:24
3

Open files

Processes with open files are the usual culprits. Display them:

lsof +f -- <mountpoint or device>

There is an advantage to using /dev/<device> rather than /mountpoint: a mountpoint will disappear after an umount -l, or it may be hidden by an overlaid mount.

fuser can also be used, but to my mind lsof has a more useful output. However fuser is useful when it comes to killing the processes causing your dramas so you can get on with your life.

List files on <mountpoint> (see caveat above):

fuser -vmM <mountpoint>

Interactively kill only processes with files open for writing:

fuser -vmMkiw <mountpoint>

After remounting read-only (mount -o remount,ro <mountpoint>), it is safe(r) to kill all remaining processes:

fuser -vmMk <mountpoint>

Mountpoints

The culprit can be the kernel itself. Another filesystem mounted on the filesystem you are trying to umount will cause grief. Check with:

mount | grep <mountpoint>/

For loopback mounts, also check the output of:

losetup -la

Anonymous inodes (Linux)

Anonymous inodes can be created by:

  • Temporary files (open with O_TMPFILE)
  • inotify watches
  • [eventfd]
  • [eventpoll]
  • [timerfd]

These are the most elusive type of pokemon, and appear in lsof's TYPE column as a_inode (which is undocumented in the lsof man page).

They won't appear in lsof +f -- /dev/<device>, so you'll need to:

lsof | grep a_inode

For killing processes holding anonymous inodes, see: List current inotify watches (pathname, PID).

1

The question how to check if NFS access a directory which is going to unmount is still unanswered.

What I have is only this:

Check if nfsd is running:

pidof nfsd

Show mounted directories by clients:

showmount -a

and showmount w/o arguments shows only client hosts even if they are off line. I assume this is a special behavior of NFS.

1

For me, the issue was that I was logged in more than once (via ssh) and on one of the logins I was at command prompt where the pwd was inside a folder subordinate to the mount-point.

protected by Community Jul 31 '18 at 23:48

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?