We have an NFS server and several clients (Ubuntu 18.04). Sometimes (rarely) a client sees corrupted data after a file was changed from another client. The server and all other clients see the file correctly.
The context is a simple everyday developer workflow situation: I edit the Python source code of some program in my IDE (PyCharm) on my client machine, then in a terminal window I SSH to a different (more performant) computer to execute the program. The symptom: null bytes are read at the end of the file, or the file appears truncated (and results in a SyntaxError in case of Python code execution.)
I suspect this is due to a bug in caching. Specifically, the affected file contents are still incorrectly seen as the old version, but all metadata is seen correctly as the new version.
Therefore, if the file was made smaller by the other client, it will be seen by our faulty client as if it had the old content, but abruptly stopped at the new size. If it was made larger by the other client, the faulty client sees the old content, but with some garbage content appended at the end, mostly zero bytes.
I have read the NFS man page on cache consistency but the problem is not with attribute caching, and there is not much in the man page about content caching perhaps because it's not handled by NFS, but a generic file system caching layer.
The attributes (inode number, size, modification times) are all correctly seen, but the content is not.
Note that this is NOT simply about the configurable attribute caching time limits. The bad file content remains there indefinitely and the client never notices that anything is wrong. I repeat, the client gets stuck with the bad content (e.g. containing spurious null bytes) for DAYS (and more) unless the cache is manually dropped.
What could be the reason for it? Is there a workaround? I know it can be temporarily fixed by dropping the caches, but it's a manual intervention and the failure can even be silent if the size happens to stay the same, just some bytes are changed on another client.
Several years on, the problem persists and there seems to be no solution anywhere (I've read FAQ's, book chapters on NFS, dug into mailing lists, read that post about how GitLab hunted down some caching bug in two weeks of debugging etc., but none of these actually address my issues.)
The most relevant info on the web comes from this mailing list discussion from 2015 but ultimately the developers claim there is no problem at all, which seems baffling to me. If this is the case, how can anyone use NFS in HPC cluster setups where one edits the code on a login node and submits it for execution on a compute node? You would regularly get issues that the compute node sees a corrupted version of the source code. More importantly, regardless of the philosophy behind a possible WONTFIX/NOBUG claim from the NFS people, what is an actionable fix in my scenario?