how to reliably determine processes using files after `umount -l` on linux

Question

The Question

After umount -l was performed on some mounted filesystem, and this filesystem remains still mounted (albeit hidden) because of some process maintaining (at least) one open file handle to some file(s) on this filesystem, how to reliably acquire a list of such processes?

The Test Setup

I performed the following steps to create such a situation and poke around in the system and try out various commands (using bach on debian jessie with a linux 3.16 kernel):

sudo -i
cd $(mktemp -d) # get empty directory to play around with
# create empty file system with one file
dd if=/dev/zero of=disk bs=1M count=4
mkfs.ext4 disk
# mount the filesytem and create a file
mkdir mounted
mount disk mounted # uses /dev/loop0 in my case, actual device my vary
touch mounted/file
# fork of a process with an open file handle to file
bash -c 'cd mounted; exec 3<>file; while true; do sleep inf; done' &
# verify that the file is open in the background
lsof mounted/file
# lazy unmount
umount -l mounted
# even remove the mountpoint 
rmdir mounted

So what command would display all processes having open some files in the hidden-mounted /dev/loop0?

Failed attempts

lsof

lsof -p <thepid>

First this requires prior knowledge of the process id (which is the information searched here), and secondly even then the open file is shown as /file with no indication that this is on the hidden mounted filesystem and not the normal root filesystem.

fuser

I could not get any useful output from fuser. I think something in the manpage eludes me.

inspecting /sys/fs/

In this particular test setup, the filesystem (ext4) and the device (loop0) is known. There it could be checked if the filesystem already changed from the hidden-mounted to the unmounted state.

if [ -e /sys/fs/ext4/loop0/ ]; then
    echo "still not unmounted";
else
    echo "finally unmounted";
fi

But this does not give a list of processes which have an open file handle.

manually inspecting /proc

In proc some information is found:

assuming that mnt_id is unique system-wide (i have no knowledge if that assumption is true!) one can make a list of all non-hidden mnt_ids across all processes:

find /proc/ -maxdepth 1 -type d -regex '/proc/[0-9]+' -exec "cat" "{}/mountinfo" ";" | cut -d " " -f 1 | sort -gu

and then make a list of all actually used mnt_ids by any open fd:

find /proc/ -regex '/proc/[0-9]+/fdinfo/[0-9]+' -exec cat "{}" ";" | grep mnt_id | cut -f 2 | sort -gu
# maybe gives some errors like 'Operation not permitted' or 'No such file or directory' because of inspecting it's own process and sub-process and possibly parallel things happening. probably this is an unreliable way of inspecting /proc.

If an sophisticated approach would be used to compare these two lists, one could determine hidden mnt_ids together with the the fd using them and process-ids of these fds. But even with that result, the last step of knowing which of these hidden mnt_ids corresponds to the hidden-mounted filesystem of interest, would be still missing.

Answer 1

Because of the way the mount point gets hidden with umount -l, there is no way to find which processes are still using affected files.

The only way to get the list is to use lsof before umount -l to grep the relevant path. Example: lsof | grep "/mountPoint/".

If you want, you can take that output to extract the PIDs and continue monitoring them.