5

The Question

After umount -l was performed on some mounted filesystem, and this filesystem remains still mounted (albeit hidden) because of some process maintaining (at least) one open file handle to some file(s) on this filesystem, how to reliably acquire a list of such processes?

The Test Setup

I performed the following steps to create such a situation and poke around in the system and try out various commands (using bach on debian jessie with a linux 3.16 kernel):

sudo -i
cd $(mktemp -d) # get empty directory to play around with
# create empty file system with one file
dd if=/dev/zero of=disk bs=1M count=4
mkfs.ext4 disk
# mount the filesytem and create a file
mkdir mounted
mount disk mounted # uses /dev/loop0 in my case, actual device my vary
touch mounted/file
# fork of a process with an open file handle to file
bash -c 'cd mounted; exec 3<>file; while true; do sleep inf; done' &
# verify that the file is open in the background
lsof mounted/file
# lazy unmount
umount -l mounted
# even remove the mountpoint 
rmdir mounted

So what command would display all processes having open some files in the hidden-mounted /dev/loop0?

Failed attempts

lsof

lsof -p <thepid>

First this requires prior knowledge of the process id (which is the information searched here), and secondly even then the open file is shown as /file with no indication that this is on the hidden mounted filesystem and not the normal root filesystem.

fuser

I could not get any useful output from fuser. I think something in the manpage eludes me.

inspecting /sys/fs/

In this particular test setup, the filesystem (ext4) and the device (loop0) is known. There it could be checked if the filesystem already changed from the hidden-mounted to the unmounted state.

if [ -e /sys/fs/ext4/loop0/ ]; then
    echo "still not unmounted";
else
    echo "finally unmounted";
fi

But this does not give a list of processes which have an open file handle.

manually inspecting /proc

In proc some information is found:

assuming that mnt_id is unique system-wide (i have no knowledge if that assumption is true!) one can make a list of all non-hidden mnt_ids across all processes:

find /proc/ -maxdepth 1 -type d -regex '/proc/[0-9]+' -exec "cat" "{}/mountinfo" ";" | cut -d " " -f 1 | sort -gu

and then make a list of all actually used mnt_ids by any open fd:

find /proc/ -regex '/proc/[0-9]+/fdinfo/[0-9]+' -exec cat "{}" ";" | grep mnt_id | cut -f 2 | sort -gu
# maybe gives some errors like 'Operation not permitted' or 'No such file or directory' because of inspecting it's own process and sub-process and possibly parallel things happening. probably this is an unreliable way of inspecting /proc.

If an sophisticated approach would be used to compare these two lists, one could determine hidden mnt_ids together with the the fd using them and process-ids of these fds. But even with that result, the last step of knowing which of these hidden mnt_ids corresponds to the hidden-mounted filesystem of interest, would be still missing.

2

Because of the way the mount point gets hidden with umount -l, there is no way to find which processes are still using affected files.

The only way to get the list is to use lsof before umount -l to grep the relevant path. Example: lsof | grep "/mountPoint/".

If you want, you can take that output to extract the PIDs and continue monitoring them.

  • For some reason I didn't think of listing processes before the umount -l command. I think with your approach, the mnt_id of the mount point could be determined, and thus monitored after umount -l was executed. this would also cover cases when, for some reason, additional processes where started between lsof and umount -l and/or additional file handles opened on the hidden-mounted filesystem after umount -l (if that's even possible?) – Max-Julian Pogner May 28 '16 at 18:03
  • I did some tests and the path visible to lsof after the umount shows on /. – Julie Pelletier May 28 '16 at 18:12
  • Yes it does, but as I understood your initial suggestion, the initial lsof would give a list of process-ids, which could at least be monitored until their orderly exit happens; or killed if they do not terminate when expected to. However, strictly speaking, this would not look at the "using files". But thanks to your comment, I had the idea to monitor the mnt_ids of the open files by the processes identified by the initial lsof -- therefore only waiting for closing file-handles would be required. Maybe you could comment if you think such approach could work? – Max-Julian Pogner May 28 '16 at 18:35
  • You wanted the list of process IDs: lsof | grep "/usr/" |cut -c10-16|sort -nu (double-check the cut positioning on your system but it should be fine as is). – Julie Pelletier May 28 '16 at 18:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.