5

I have a /dev/md127 mdadm RAID 0 array that some time ago was mounted as /mnt/storage1. At some point I opened a bash session and changed CWD to /mnt/storage1 and the bash session is still active. I then decided to unmount the array and destroy it so I did:

/# umount /mnt/storage1
Device or resource busy msg
/# umount -l /mnt/storage1
(Succeeded)
/# rmdir /mnt/storage1
(Succeeded)

I confirmed that the /mnt/storage1 has been deleted. mount doesn't show /dev/md127 as mounted. Still, the bash session I mentioned still has /mnt/storage1 as its working directory:

/mnt/storage1# _

Now, when I try to stop and destroy the /dev/md127 array I get in return:

/# mdadm --stop /dev/md127
mdadm: Cannot get exclusive access to /dev/md127:Perhaps a running process, mounted filesystem or active volume group?

lsof doesn't list any files as being still open on either /dev/md127 or /mnt/storage1

/# lsof |grep storage1
/# (No results)
/# lsof |grep md127
/# (No results)

I even tried to list files opened by the bash process that is still in the /mnt/storage1 directory but without success (and yes, 3172 is correct PID of the bash process)

/# lsof -p 3172
COMMAND  PID USER   FD   TYPE DEVICE SIZE/OFF     NODE NAME
bash    3172 root  cwd    DIR   0,40       40      256 /
bash    3172 root  rtd    DIR    9,0     4096        2 /
bash    3172 root  txt    REG    9,0  1037528 10485776 /bin/bash
bash    3172 root  mem    REG    9,0    47600  1310937 /lib/x86_64-linux-gnu/libnss_files-2.23.so
bash    3172 root  mem    REG    9,0    47648  1310851 /lib/x86_64-linux-gnu/libnss_nis-2.23.so
bash    3172 root  mem    REG    9,0    93128  1310763 /lib/x86_64-linux-gnu/libnsl-2.23.so
bash    3172 root  mem    REG    9,0    35688  1310755 /lib/x86_64-linux-gnu/libnss_compat-2.23.so
bash    3172 root  mem    REG    9,0  2981280 16522333 /usr/lib/locale/locale-archive
bash    3172 root  mem    REG    9,0  1864888  1311188 /lib/x86_64-linux-gnu/libc-2.23.so
bash    3172 root  mem    REG    9,0    14608  1311189 /lib/x86_64-linux-gnu/libdl-2.23.so
bash    3172 root  mem    REG    9,0   167240  1311191 /lib/x86_64-linux-gnu/libtinfo.so.5.9
bash    3172 root  mem    REG    9,0   162632  1311181 /lib/x86_64-linux-gnu/ld-2.23.so
bash    3172 root  mem    REG    9,0    26258 16523837 /usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache
bash    3172 root    0u   CHR  136,0      0t0        3 /dev/pts/0
bash    3172 root    1u   CHR  136,0      0t0        3 /dev/pts/0
bash    3172 root    2u   CHR  136,0      0t0        3 /dev/pts/0
bash    3172 root  255u   CHR  136,0      0t0        3 /dev/pts/0

I even tried to get CWD of the bash process but this has given wrong(?) result:

/# pwdx 3172
3172: /

Let's for a second assume that I don't know which process is preventing me from stopping the array. How can I identify it?

This question is related to https://superuser.com/questions/471327/how-to-force-mdadm-to-stop-raid5-array - the problem has been bugging me for several years and now once it has happened to me again I want to solve it properly. The bash session is still open and ready to test your answer :-)

Please note that this question is not about how to stop the array but how to identify the process that is still using a file from the array / prevents it from being destroyed.

  • 4
    I think the lazy unmount is the culprit. The filesystem is still mounted (because there are open files on it) but it has no location in the directory tree so tools that process paths of open files don't find it. – Gilles Feb 16 '17 at 23:25
  • 1
    I'm not able to test this out right now but does this help? blogs.janestreet.com/… – roaima Feb 16 '17 at 23:36
  • @roaima Yes, it might be good starting point. As mentioned in the article the path /proc/$pid/cwd is truncated and may give wrong impression (in my case it seemed to point to /, the /mnt/storage1 part has been chopped off leaving just /). readlink -f /proc/$pid/cwd doesn't produce error or any warning and seems to succeed, [ -e/-f /proc/$pid/cwd ] behaves as if the path still existed. However when readlink is executed from the lazily unmounted directory (readlink -f .) it does show (unreachable)/. – matt Feb 17 '17 at 9:26
2

As per Finding references to lazily unmounted filesystems page, the lsof tool doesn’t appear to list non-absolute paths (lsof’s output is irregular) and what's worse, it doesn’t list other filesystem dependencies like memory maps.

As for workaround, you'll have to look at /proc/*/maps which presents the memory mappings which belong to each process to indicate the kind of mapping and, if it’s a file or path. However like lsof, the absolute path is unavailable if the filesystem hosting the file was lazy unmounted.

Here is the suggested script:

!/bin/bash
cat /proc/*/maps 
  | awk '{print $6}'
  | grep -v '^/'         # remove absolute paths
  | grep -v '^$' 
  | grep -v '(deleted)' 
  | grep -v '^.vdso.$' 
  | grep -v '^.heap.$' 
  | grep -v '^.stack.$' 
  | grep -v '^.vsyscall.$' 
  | grep -v '^socket:$'

which can help to remove known false positives.

Further more, you can also check in /proc/X/fd/* and /proc/X/cwd.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.