0

There are 3 servers where NFS (ver.4) partition is mounted. Servers A & B are just fine, however on server C partition is mounted not properly. Changes are made on servers A & B are not propagated to server C and vise-versa. mount on server C shows partition as mounted:

mount |grep snapshots
swdvnasots0014.nam.nsroot.net:/swdicgv000803/icgqt000803_c153904 on /opt/gicapods/snapshots type nfs4 (rw,relatime,vers=4.0,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=xxx.xxx.148.149,local_lock=none,addr=10.47.15.114)

However df shows it only on direct query

df -h|grep snapshots
 
df -h /opt/gicapods/snapshots/
Filesystem                                                        Size  Used Avail Use% 
Mounted on
swdvnasots0014.nam.nsroot.net:/swdicgv000803/icgqt000803_c153904  400G  163G  223G  43% /opt/gicapods/snapshots

The line from fstab

grep snapshots /etc/fstab
swdvnasots0014.nam.nsroot.net:/swdicgv000803/icgqt000803_c153904  /opt/gicapods/snapshots nfs defaults        0 0

I searched /var/log/messages since server's last reboot, but didn't find anything related.

Pls advise what could be a root cause here. It there any way to prevent this from happening again?

UPD:

grep -e snapshots -e swdvnasots0014 -e swdicgv000803 /proc/self/mountinfo /proc/self/mounts /proc/mounts /etc/mtab 
/proc/self/mountinfo:187 49 0:42 / /opt/gicapods/snapshots rw,relatime shared:164 - nfs4 swdvnasots0014.nam.nsroot.net:/swdicgv000803/icgqt000803_c153904 rw,vers=4.0,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=xxx.xxx.148.149,local_lock=none,addr=10.47.15.114 
/proc/self/mounts:swdvnasots0014.nam.nsroot.net:/swdicgv000803/icgqt000803_c153904 /opt/gicapods/snapshots nfs4 rw,relatime,vers=4.0,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=xxx.xxx.148.149,local_lock=none,addr=10.47.15.114 0 0
/proc/mounts:swdvnasots0014.nam.nsroot.net:/swdicgv000803/icgqt000803_c153904 /opt/gicapods/snapshots nfs4 rw,relatime,vers=4.0,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=xxx.xxx.148.149,local_lock=none,addr=10.47.15.114 0 0
/etc/mtab:swdvnasots0014.nam.nsroot.net:/swdicgv000803/icgqt000803_c153904 /opt/gicapods/snapshots nfs4 rw,relatime,vers=4.0,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=xxx.xxx.148.149,local_lock=none,addr=10.47.15.114 0 0
6
  • Try a plain df -h, then it might become clear why it's not being extracted by the grep.
    – Bib
    Commented Oct 19, 2022 at 9:06
  • It is not presented in df -h output Commented Oct 19, 2022 at 9:24
  • 1. What makes you think the disk is not mounted even though it appears in mount? 2. What happens when you write data to the disk from machine C - do you see it in A and B? 3. Is the disk mounted with the same mount options on all 3 machines?
    – aviro
    Commented Oct 19, 2022 at 12:13
  • @aviro 1. Data is not propagated 2. No 3. Yes Commented Oct 19, 2022 at 13:39
  • What is the output of the following command? grep -e snapshots -e swdvnasots0014 -e swdicgv000803 /proc/self/mountinfo /proc/self/mounts /proc/mounts /etc/mtab Please edit your question and add there the output of your command (not in the comments)
    – aviro
    Commented Oct 19, 2022 at 14:23

1 Answer 1

0

It's difficult next to impossible to just provide a root cause for such a case, but I will provide few tips for debugging and investigation, since it's difficult to do that in the comments section. You can add the output of different actions to your original question, and I will try to follow by editing the answer, until hopefully we can reach the root cause.

  1. The first important thing is to express the problem in a clear and different manner. "Disk is not mounted" is different than "changes are not propagated". We clearly see that the disk is mounted, from all the mount related files and commands.

  2. Now, you say the changes are not propagated. What's the initial state of the mount? Is the content of the mount completely different between machines A/B and C? Or are there some common files? Does it look like it was the same mount at some point (until "changes stopped being propagated"), or is the content completely different?

  3. Another thing is to carefully check if indeed all of the mount options and the sources are exactly the same on all three machines. It's easy to overlook some differences and type-o's, especially when the name of the mount is so complicated and contains different numbers. You might used a different server or disk on host C, and the difference might be even one small character that changes everything. So you should run the following command on all three machines, and check if it's entirely the same. Check every character to ensure it's identical. Maybe it's a different server or IP?

mount |grep snapshot
  1. When you compare the disks, do you compare the same folders? Is there a difference between the main mount folder and the subfolders? Try to create a file on machines C and check the file is indeed mounted at the same place, but it doesn't exist on hosts A and C.
touch /opt/gicapods/snapshots/testfile
df /opt/gicapods/snapshots/testfile
  1. Right after you make the test in the previous step, check if there's an open port to the NFS server (don't linger, because those ports are closed after some idle time, and are reopened by the kernel the next time you access the mount).
ss |grep <IP of the NFS file server>
  1. Also check if there's any network communication between host C and the server, by using tcpdump on one terminal, and trying to make some operations on another terminal (ls, find, read, write etc). When you're doing the operation on the mount, check if tcpdump outputs anything.
tcpdump -s0 -i any host swdvnasots0014
  1. Try to mount the disk manually on another mount point on host C. What do you see? Is the content inside this new mount point identical to what you see on /opt/gicapods/snapshots on host C, or on hosts A/B? What happens when you make a change in this new mount point, do you see the changes propagated to the original mount point on host C, or on hosts A/B?

Hopefully by this point you'll have a few leads. If you found the answer during the investigation, please update it in your question. If not, just add the results of the different tests to the questions so we could continue. Good luck!

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .