237

When running umount /path I get:

umount: /path: device is busy.

The filesystem is huge, so lsof +D /path is not a realistic option.

lsof /path, lsof +f -- /path, and fuser /path all return nothing. fuser -v /path gives:

                  USER        PID ACCESS COMMAND
/path:            root     kernel mount /path

which is normal for all unused mounted file systems.

umount -l and umount -f is not good enough for my situation.

How do I figure out why the kernel thinks this filesystem is busy?

Improve this question
5
  • 29
    Is your shell's current directory on the mountpoint path?
    – LawrenceC
    Jun 15, 2011 at 11:49
  • 2
    No. Then fuser would say so.
    – Ole Tange
    Jun 15, 2011 at 11:58
  • 20
    You actually want fuser -vm /path ...
    – derobert
    Oct 23, 2012 at 17:23
  • 11
    For umount --force will try harder to unmount and -v or -vvv even will reaveal more what is the problem with mount. So try: umount -vvv --force /babdmount
    – gaoithe
    Dec 4, 2015 at 11:50
  • @derobert: Which in my case revealed an active swapfile and was the solution to my version of the "device busy" problem. Thank you!
    – DevSolar
    Oct 6, 2021 at 9:23

15 Answers 15

Reset to default
183

It seems the cause for my issue was the nfs-kernel-server was exporting the directory. The nfs-kernel-server probably goes behind the normal open files and thus is not listed by lsof and fuser.

When I stopped the nfs-kernel-server I could umount the directory.

I have made a page with examples of all solutions so far here: http://oletange.blogspot.com/2012/04/umount-device-is-busy-why.html

Improve this answer
9
  • 70
    Thank you for answering your own question instead of abandoning it upon implementing your solution. Your answer helped me sort out a similarly exported NFS share.
    – Jeff Welling
    Dec 15, 2011 at 5:44
  • 11
    This same issue can also occur if you've set up loopback devices on the filesystem - for example if /dev/loop0 is backed by a file in /path.
    – BCran
    Feb 22, 2012 at 13:25
  • 2
    I had to sudo service samba stop first, your answer really helped out !
    – malat
    Jun 10, 2014 at 7:38
  • 1
    This post reminded me that I had the nfs service running after several hours of trying to figure this out. In RHEL6/CentOS6, use sudo service nfs stop and you may (not) need to also do sudo exportfs -u to unexport. Remember to then sudo exportfs -r and sudo service nfs start to re-export and restart the service.
    – code_dredd
    May 3, 2016 at 0:45
  • 2
    In my case it was not necessary to stop the nfs server, just exportfs -u the directory in question.
    – Law29
    Nov 27, 2016 at 13:34
51

To add to BruceCran's comment above, the cause for my manifestation of this problem just now was a stale loopback mount. I'd already checked the output of fuser -vm <mountpoint>/lsof +D <mountpoint>, mount and cat /proc/mounts, checked whether some old nfs-kernel-server was running, turned off quotas, attempted (but failed) a umount -f <mountpoint> and all but resigned myself to abandoning 924 days' uptime before finally checking the output of losetup and finding two stale configured-but-not-mounted loopbacks:

parsley:/mnt# cat /proc/mounts 
rootfs / rootfs rw 0 0
none /sys sysfs rw,nosuid,nodev,noexec 0 0
none /proc proc rw,nosuid,nodev,noexec 0 0
udev /dev tmpfs rw,size=10240k,mode=755 0 0
/dev/mapper/stuff-root / ext3 rw,errors=remount-ro,data=ordered 0 0
tmpfs /lib/init/rw tmpfs rw,nosuid,mode=755 0 0
usbfs /proc/bus/usb usbfs rw,nosuid,nodev,noexec 0 0
tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0
devpts /dev/pts devpts rw,nosuid,noexec,gid=5,mode=620 0 0
fusectl /sys/fs/fuse/connections fusectl rw 0 0
/dev/dm-2 /mnt/big ext3 rw,errors=remount-ro,data=ordered,jqfmt=vfsv0,usrjquota=aquota.user 0 0

then

parsley:/mnt# fuser -vm /mnt/big/
parsley:/mnt# lsof +D big
parsley:/mnt# umount -f /mnt/big/
umount2: Device or resource busy
umount: /mnt/big: device is busy
umount2: Device or resource busy
umount: /mnt/big: device is busy

parsley:/mnt# losetup -a    
/dev/loop0: [fd02]:59 (/mnt/big/dot-dropbox.ext2)
/dev/loop1: [fd02]:59 (/mnt/big/dot-dropbox.ext2)

parsley:/mnt# losetup -d /dev/loop0
parsley:/mnt# losetup -d /dev/loop1
parsley:/mnt# losetup -a
parsley:/mnt# umount big/
parsley:/mnt#

A Gentoo forum post also lists swapfiles as a potential culprit; although swapping to files is probably pretty rare these days, it can't hurt to check the output of cat /proc/swaps. I'm not sure whether quotas could ever prevent an unmount — I was clutching at straws.

Improve this answer
3
  • 15
    All 924 days uptime means is that you need to update your kernel patches :-)
    – w00t
    Jun 25, 2012 at 13:39
  • 1
    +1 for mentioning swap files, they do block unmounting, and are pretty much undetectable if you are not checking them directly.
    – P.Péter
    Sep 13, 2019 at 11:53
  • I also had a stale loop0, but losetup -d /dev/loop0 would NOT actually detach it. running -a still showed it there; never got it to go away.
    – bparker
    Oct 10, 2021 at 19:26
33

Instead of using lsof to crawl through the file system, just use the total list of open files and grep it. I find this returns must faster, although it's less accurate. It should get the job done.

lsof | grep '/path'
Improve this answer
4
  • 1
    lsof /path looks through the path only.
    – Ole Tange
    Jun 15, 2011 at 11:58
  • 11
    I didn't say lsof /path, I said lsof | grep '/path'. The difference is that lsof with no arguments shows all open files using some sort of cache table, and grep is very fast at searching through it. The things you tried with lsof make it scan through the file system which takes a long time.
    – Caleb
    Jun 15, 2011 at 12:02
  • 2
    Like I said: lsof /path looks at the path only. It does not look at every single file. It is often much faster than lsof | grep /path (in my unscientific test it was 20 times faster YMMV) since it does not look at all open files but only the files for that path.
    – Ole Tange
    Jun 16, 2011 at 10:10
  • 3
    I'm not sure of the what the technical difference is, but while investigating a stale NFS mount, lsof /path yielded nothing whereas lsof | grep /path showed me the process that was holding open files and preventing me from unmounting the volume.
    – dpw
    Sep 27, 2017 at 16:34
24

For me, the offending process was a daemon running in a chroot. Because it was in a chroot, lsof and fuser wouldn't find it.

If you suspect you have something left running in a chroot, sudo ls -l /proc/*/root | grep chroot will find the culprit (replace "chroot" with the path to the chroot).

Improve this answer
2
  • 1
    Nice, and in FreeBSD I did this: sudo ls -l /proc/*/status | grep HOST where HOST is the hostname of the jail
    – JGurtz
    Jul 24, 2013 at 22:31
  • 1
    On my system (Mint Qiana) lsof /mountpoint and fuser /mountpoint both find a process even if chrooted.
    – Ole Tange
    Dec 21, 2014 at 17:20
23

Open files

Processes with open files are the usual culprits. Display them:

lsof +f -- <mountpoint or device>

There is an advantage to using /dev/<device> rather than /mountpoint: a mountpoint will disappear after an umount -l, or it may be hidden by an overlaid mount.

fuser can also be used, but to my mind lsof has a more useful output. However fuser is useful when it comes to killing the processes causing your dramas so you can get on with your life.

List files on <mountpoint> (see caveat above):

fuser -vmM <mountpoint>

Interactively kill only processes with files open for writing:

fuser -vmMkiw <mountpoint>

After remounting read-only (mount -o remount,ro <mountpoint>), it is safe(r) to kill all remaining processes:

fuser -vmMk <mountpoint>

Mountpoints

The culprit can be the kernel itself. Another filesystem mounted on the filesystem you are trying to umount will cause grief. Check with:

mount | grep <mountpoint>/

For loopback mounts, also check the output of:

losetup -la

Anonymous inodes (Linux)

Anonymous inodes can be created by:

  • Temporary files (open with O_TMPFILE)
  • inotify watches
  • [eventfd]
  • [eventpoll]
  • [timerfd]

These are the most elusive type of pokemon, and appear in lsof's TYPE column as a_inode (which is undocumented in the lsof man page).

They won't appear in lsof +f -- /dev/<device>, so you'll need to:

lsof | grep a_inode

For killing processes holding anonymous inodes, see: List current inotify watches (pathname, PID).

Improve this answer
2
  • Your fuser suggestion was very helpful when booted into the Arch Linux installation media which doesn't come with lsof by default.
    – HellaMad
    May 26, 2021 at 8:02
  • this answer is fantastic. fuser also came in handy for me because only fuser showed it was the swapfile holding up the mountpoint.
    – Gaia
    Oct 21, 2021 at 5:48
7

For fuser to report on the PIDs holding a mount open you have to use -m

fuser -m /path
Improve this answer
3
  • 2
    True, but irrelevant: lsof /path provides the same list of PIDs as fuser -m /path.
    – Gilles 'SO- stop being evil'
    Jun 16, 2011 at 7:55
  • fuser -M /path will check whether /path is a mountpoint.
    – user3804598
    Jun 9, 2019 at 13:22
  • This is the only thing that actually worked for me. umount -f did nothing. lsof didn't work either. The only weird thing is it prints a c after the PID but just ignore that.
    – Timmmm
    Jun 9 at 10:53
6

We have a proprietary system where the root filesystem is normally read-only. Occasionally, when files have to be copied over, it is remounted read-write:

mount -oremount,rw /

And then remounted back:

mount -oremount,ro /

This time however, mount kept giving the mount: / is busy error. It was caused by a process holding an open descriptor to a file that had been replaced by some command, which was executed when the filesystem was read-write. The important line from lsof -- / output happens to be (names have been changed):

replicate  1719 admin DEL REG 8,5  204394 /opt/gns/lib/replicate/modules/md_es.so

Notice the DEL in the output. Simply restarting the process holding on to the deleted file resolved the issue.

Improve this answer
1
  • 3
    So the summary is: process having a file opened that was removed. Good input.
    – Ole Tange
    May 2, 2016 at 14:03
4

I had this issue, and it turned out that there were active screen sessions in the background I didn't know about. I connected to the other active screen session and its shell wasn't even currently sitting in the mounted directory. Killing those other shell sessions fixed the issue for me.

Just thought I'd share my resolution.

Improve this answer
4

lsof and fuser didn't give me anything either.

After a process of renaming all possible directories to .old and rebooting the system every time after I made changes I found one particular directory (relating to postfix) that was responsible.

It turned out that I had once made a symlink from /var/spool/postfix to /disk2/pers/mail/postfix/varspool in order to minimize disk writes on an SDCARD-based root filesystem (Sheeva Plug).

With this symlink, even after stopping the postfix and dovecot services (both ps aux as well as netstat -tuanp didn't show anything related) I was not able to unmount /disk2/pers.

When I removed the symlink and updated the postfix and dovecot config files to point directly to the new dirs on /disk2/pers/ I was able to successfully stop the services and unmount the directory.

Next time I will look more closely at the output of:

ls -lR /var | grep ^l | grep disk2

The above command will recursively list all symbolic links in a directory tree (here starting at /var) and filter out those names that point to a specific target mount point (here disk2).

Improve this answer
1
  • find /var -type l -lname '*disk2*' -printf '%p -> %l\n' does the same as the ls pipe
    – go2null
    Jan 3, 2021 at 6:26
3

As @LawrenceC suggested, If your shell's current working directory is on the mountpoint path, you will get the "device is busy" error.

ubuntu@myip:~/efs$ sudo umount ~/efs
umount.nfs4: /home/ubuntu/efs: device is busy

cd to a location other than the mountpoint to resolve the error.

Improve this answer
3
  • OP expressely stated that his current working directory was not on the mount path.
    – guntbert
    Jan 29, 2020 at 21:26
  • 1
    @guntbert since comments are ephemeral and neither the question or answers mention the issue of whether the current working directory is on the mount path, I added it as an explicit answer.
    – enharmonic
    Jan 29, 2020 at 21:39
  • 2
    @guntbert you are right, however it is not unlikely that this is the problem for someone coming here from a search engine (like it was for me).
    – Newbyte
    May 11, 2020 at 19:38
2

Today the problem was an open socket (specifically tmux):

mount /mnt/disk
export TMPDIR=/mnt/disk
tmux
<<detatch>>
umount /mnt/disk
umount: /mnt/disk: device is busy.
lsof | grep /mnt/disk
tmux      20885             root    6u     unix 0xffff880022346300        0t0    3215201 /mnt/disk/tmux-0/default
Improve this answer
1
  • This is really absurd, but yea tmux was culprit for me too!
    – Xolve
    May 15, 2021 at 13:22
2

I have a couple of bind and overlay mounts under my mount that were blocking me, check the tab completion for the mount-point you want to unmount. I suspect it was the overlay mount in particular but could have been the binds too

Improve this answer
1

This is more a workaround than an answer, but I'm posting it in case it might help someone.

In my case I was trying to modify the LVM as I wanted to make the /var partition bigger, so I needed to umount it. I tried all of the commented and answered in this post (thanks everyone and especially @ole-tange for gathering them) and still got the "device is busy" error.

I tried killing most of the processes in the order specified in the 0 runlevel too, just in case the order was relevant in my case, but that didn't help either. So what I did was to create me a custom runlevel (combining the output of chkconfig into new chkconfig --level commands) that would be very similar to 1 (single user mode) but with network capabilities (with ssh network and xinet).

As I was using redhat, runlevel 4 is marked as "unused/user defined", so I used that one, and run init 4 In my case this was ok as I needed to reboot the server in any case, but probably that will be the case of anyone tweaking the disks.

Improve this answer
0

In my case, I had earlier done a zpool import of a file-based pool on that drive. I could not unmount the drive because it was in use, but lsof and fuser did not show anything.

The solution was to do sudo zpool export mypool and then unmount.

Improve this answer
0

In my case it was docker that was holding the file. I am running ZFS, and after lazy umounting the volume, I got an error message:

mount: /tank: /dev/zd112 already mounted or mount point busy.

The investigation

fuser /dev/zd112
Cannot stat file /proc/32130/fd/4: Stale file handle
Cannot stat file /proc/32130/fd/5: Stale file handle
Cannot stat file /proc/32130/fd/6: Stale file handle
Cannot stat file /proc/32130/fd/7: Stale file handle
Cannot stat file /proc/32130/fd/11: Stale file handle

Running ps aux |grep 32130 results:

999      32130  4.1  1.2 10784420 1241840 ?    Ssl  Jun20 10055:41 mysqld

Which is not a mysql run in the same userspace, but within a container.

Running docker ps |grep mysql yields

3c6c907a28db        mysql:5.7                                                                  "docker-entrypoint.s…"   15 months ago       Up 5 months         33060/tcp, 0.0.0.0:3336->3306/tcp   zabbix_db_1_5e72b12e41d2

Next you can run:

docker stop 3c6c907a28db
Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .