3

I have an automated script that moves data files from the computer that they're collected on to a central NFS server for more general access. I've recently been seeing about 1% of files failing to be moved, with mv reporting an error about the file already existing:

mv: cannot create regular file `/mnt/data/2015/20150410_004130.dat': File exists

The thing is that until immediately before, the file doesn't exist, and after mv tries to create it, it exists, but with size 0, permissions 0000, and a date back in the 1970s (the exact date seems to increment slightly each time this happens).

I updated my script to run mv using strace, and the cases that fail look exactly like the ones that succeed until the actual failure occurs:

  1. try to rename directly. Fails (Invalid cross-device link)
  2. unlink target. Fails (No such file or directory)
  3. open source file for reading
  4. open target file with options O_WRONLY|O_CREAT|O_EXCL, perms 0600. Usually works, occasionally fails, claiming "File exists"

I'm fairly confident that nothing else should be trying to create this file, although there is another script that is processing data files (actually moving them to another server) once they're finished.

This behavior seems to have started about the same time I updated from Ubuntu 10.04 to Ubuntu 14.04 on the NFS server (client configuration is unchanged).

  • Are there .nfs* files in the directory? Could there be a file by that name which is still open on an NFS client, but deleted? Are you using the userland or kernel NFS server? Can you trace accesses on the server side? – Gilles Apr 10 '15 at 21:57
  • @Gilles each NFS directory is exported to only one client, and since the files are named by date, there shouldn't really be any issues with re-use. I believe Ubuntu's NFS server is in the kernel. I haven't actually tried, but my impression from reading around is that strace won't help very much server side when the server is in the kernel? Also not a good time to debug right now as the error rate has dropped a lot for some reason (to <1/day right now). – arcticmac Apr 12 '15 at 4:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.