4

running RHEL 7.9, on LAN using a cisco switch, and cat6 wired cabling. When there is any kind of network issue, and sometimes not a network issue from what I can tell, linux will hang because of NFS, on an NFS client, why?

For example

  • with /etc/nfs.conf and /etc/sysconfig/nfs unmodified from default setup, NFS vers=4.1 happens fairly easily.
  • server A exports /data
  • server B mount /data from Server A
  • if there is any kind of network hangup, simply unplug a cat6 cable of Server A or B and reconnect... the network will reconnect and things like SSH will work between Servers A & B either way but server B as the NFS client will experience the following types of problems that I do not know how to correct:
    • NFS stale handle for /data
    • doing an ls / will hang in the terminal window until a ctrl-c is done
    • doing a df -h will hang in the terminal window until a ctrl-c is done
    • doing a umount /data will also hang
    • it seems like the only reliable fix is to reboot the NFS client

Can someone explain why such problems happen? Specifically the hanging of doing things like ls and df ? And there is any kind of instructional on how to troubleshoot NFS? I feel like I am working with a black box when it stops working the only fix is a reboot.

Improve this question

3 Answers 3

Reset to default
2

Next time you have this problem, look into /proc/fs/nfsfs/volumes on server B. You should find there a list of mounted NFS filesystems, listing a FSID for each. Note the FSIDs of the hanging mount(s). Then reboot the client, mount the NFS volumes and check again. The FSIDs should have stayed the same: if they have changed, that would be the cause.

If your FSIDs keep changing, what is the actual filesystem type of the exported /data on server A that does the exporting?

If that filesystem does not have a real UUID, or the device number of the device holding the filesystem may vary (because of hot-plugging or admin actions on server A). If there are no valid sources for a persistent FSID on server A for whatever reason, server A might have to generate one dynamically.

If the server B sees a different FSID than before when it attempts to reconnect to server A, it assumes the NFS share the server B is reconnecting to is not quite the same as it was before, and any cached data for the old share does not necessarily apply for the new one.

That puts server B in a difficult situation: any open file handles and cached data refer specifically to the old share which is apparently nowhere to be found. Just blindly applying them to the new share might be a mistake that causes loss of data. And the kernel will never willfully lose user's data without being explicitly told to.

If the kernel has cached data waiting to be written to the old share, then a normal umount /data on server B will indeed hang - but umount -l /data should work. Unfortunately, it will only allow you a cleaner shutdown - remount of a share that has been unmounted with umount -l may not be possible, unless all processes holding references to the unmounted filesystem stop doing so first.

If you have a NFS share that has no fixed FSID, you might have to add a fsid=<number>|root|<uuid> option to /etc/exports on server A, to specify a fixed FSID (either a valid UUID, or a single small integer, for legacy compatibility).

Once server B sees that the NFS shares on server A have the same FSIDs on reconnection as on the initial connection, it should be able to proceed with reconnection automatically.

Improve this answer
1
  • I use XFS everywhere, disks are behind a raid controller and always mounted via UUID in fstab. Thanks, this is something to go on.
    – ron
    Commented Oct 31, 2022 at 14:37
1

That is kind of the nature of NFS. It was that way in the 90s when I started on Unix.

Have you tried mounting with the -o soft option? See if that helps.

Here's from the man page for nfs:

       soft / hard    Determines  the  recovery behavior of the NFS client after an NFS request times out.  If neither option
                      is specified (or if the hard option is specified), NFS requests are retried indefinitely.  If the  soft
                      option  is  specified, then the NFS client fails an NFS request after retrans retransmissions have been
                      sent, causing the NFS client to return an error to the calling application.

                      NB: A so-called "soft" timeout can cause silent data corruption in certain cases. As such, use the soft
                      option  only  when  client responsiveness is more important than data integrity.  Using NFS over TCP or
                      increasing the value of the retrans option may mitigate some of the risks of using the soft option.
Improve this answer
2
  • Assuming that the drive is already mounted you mean, unmount it and mount it back with that option?
    – VaTo
    Commented Mar 6 at 22:17
  • i get device is busy, becuase my other commands are hanging lol
    – cyford
    Commented Jul 31 at 17:38
1

You can sometimes recover it using:

 mount -o remount /nfs/mountpoint

If that fails then can force the unmount. NOTE that there is a risk of corruption by doing this - probably same risk as rebooting though.

umount -f /nfs/mountpoint
Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .