5

For some time I've been experiencing a strange issue with NFS where a seemingly random subset of directories (always the same ones) under / consistently show up with stale file handles immediately after NFS mount.

I've been able to correct the problem by explicitly exporting the seemingly-random set of problem directories, but I'd like to see if I can fix things more completely so I don't have to occasionally add random dirs to the export table.

Below, I mount a filesystem, show that there are no open file handles, run ls, and rerun lsof. Empty lines added between commands for clarity:

# mount -t nfs -o vers=4,noac,hard,intr 192.168.0.2:/ /nfs -vvv
mount.nfs: trying text-based options 'vers=4,noac,hard,intr,addr=192.168.0.2,clientaddr=192.168.0.4'
192.168.0.2:/ on /nfs type nfs (rw,vers=4,noac,hard,intr)

# lsof | grep /nfs

# ls -lh /nfs
ls: cannot access /nfs/usr: Stale file handle
ls: cannot access /nfs/root: Stale file handle
ls: cannot access /nfs/etc: Stale file handle
ls: cannot access /nfs/home: Stale file handle
lrwxrwxrwx   1 root root       7 Mar 27  2017 bin -> usr/bin
drwxr-xr-x   6 root root     16K Jan  1  1970 boot
drwxr-xr-x 438 i336 users    36K Feb 28 12:12 data
drwxr-xr-x   2 root root    4.0K Mar 14  2016 dev
d?????????   ? ?    ?          ?            ? etc
d?????????   ? ?    ?          ?            ? home
lrwxrwxrwx   1 root root       7 Mar 27  2017 lib -> usr/lib
lrwxrwxrwx   1 root root       7 Mar 27  2017 lib64 -> usr/lib
drwxr-xr-x  15 root root    4.0K Oct 15 15:51 mnt
drwxr-xr-x   2 root root    4.0K Aug  9  2017 nfs
drwxr-xr-x  14 root root    4.0K Jan 28 17:00 opt
dr-xr-xr-x   2 root root    4.0K Mar 14  2016 proc
d?????????   ? ?    ?          ?            ? root
drwxr-xr-x   2 root root    4.0K Mar 14  2016 run
lrwxrwxrwx   1 root root       7 Mar 27  2017 sbin -> usr/bin
drwxr-xr-x   6 root root    4.0K Jun 22  2016 srv
dr-xr-xr-x   2 root root    4.0K Mar 14  2016 sys
drwxrwxrwt   2 root root    4.0K Dec 10  2016 tmp
d?????????   ? ?    ?          ?            ? usr
drwxr-xr-x  15 root root    4.0K May 24  2017 var

# lsof | grep /nfs

#

The subdirectories in question are not mountpoints; they seem completely normal:

$ ls -dlh /usr /root /etc /home
drwxr-xr-x 123 root root  12K Mar  3 13:34 /etc
drwxr-xr-x   7 root root 4.0K Jul 28  2017 /home
drwxrwxrwx  32 root root 4.0K Mar  3 13:55 /root
drwxr-xr-x  15 root root 4.0K Feb 24 17:48 /usr

There are no related errors in syslog about these directories. The only info that does show up mentions a different set of directories:

... rpc.mountd[10080]: Cannot export /proc, possibly unsupported filesystem or fsid= required
... rpc.mountd[10080]: Cannot export /dev, possibly unsupported filesystem or fsid= required
... rpc.mountd[10080]: Cannot export /sys, possibly unsupported filesystem or fsid= required
... rpc.mountd[10080]: Cannot export /tmp, possibly unsupported filesystem or fsid= required
... rpc.mountd[10080]: Cannot export /run, possibly unsupported filesystem or fsid= required

Here's what /etc/exports currently looks like:

/ *(rw,subtree_check,no_root_squash,nohide,crossmnt,fsid=0,sync)

The server side is running Arch Linux and currently on kernel 4.10.3.

The client side is Slackware 14.1 with kernel 4.1.6.

  • Regarding the bounty (sorry it's not a lot 😃) - I'm looking for help/ideas on what I might be able to try and poke in order to figure out what's going on here. My own efforts fairly promptly hit a brick wall when I discovered nfsd couldn't be straced as it's a kernel process. Maybe there's something I can do with BPF or similar...? Or perhaps Wireshark might come in handy? – i336_ Mar 6 '18 at 2:03
  • Experienced a slightly similar issue during this week and which seem to be caused by the NFS server. Had to reboot the NFS server which resolved it finally. Does the logs on Arch Linux (4.10.3.) show something suspicious? – U880D Mar 6 '18 at 16:39
  • 1
    I would consider reviewing how the / directory is exported on the NFS server. Moreover, if there is just a few directories you are trying to access, consider mounting them separately. Alternatively, if you need to access files under /home, for instance, try exporting and mounting that as an experiment. I should say, I have no idea what the issue is. However, just "messing around" can sometimes yield useful insights. Of course, you can try these things I've suggested and come up with nothing anyway. – Mark Mar 6 '18 at 23:16
  • Did you read this: serverfault.com/questions/617610/… ? – thomas Mar 7 '18 at 15:12
  • 1
    @i336_ I agree that doing so (testing another kernel -- note that I suggested another client also, since in your example the delta between both kernels is not small) does not look like sexy, but it depends on your goal: either to get the real final word on pinpointing precisely what created the problem (this is the research task that may not be bounded by time) or find the quickest solution to the problem (a pragmatic engineer task) in order to go forward with other stuff. Both are sensible but will not go the same route nor for the same amount of time and energy. – Patrick Mevzek Mar 9 '18 at 1:45
0

Your exports file looks abnormal for NFS 4:

/ *(rw,subtree_check,no_root_squash,nohide,crossmnt,fsid=0,sync)

Instead I think you need to follow the Arch Linux instructions for that fsid=0 line. It declares a special export, ‘the so-called NFS root.’

Then declare your own exports on subsequent lines as the instructions show. You can export the server’s root file system — not to be confused with the NFS root — as shown in this old Gentoo post.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.