6

problem

First, I log in to a new workstation.

myAcct@ws5: mkdir /users/adminAcct/foo/parentDir/childDir
mkdir: cannot create directory ‘/users/adminAcct/foo/parentDir/childDir’: Permission denied

A Permission denied error, even though the childDir exists.

If I access childDir, this error changes. I can do this in iPython with os.path.isdir() or in the shell with ls.

myAcct@ws5: ls /users/adminAcct/foo/parentDir/childDir
file.cfg
myAcct@ws5: mkdir /users/adminAcct/foo/parentDir/childDir
mkdir: cannot create directory ‘/users/adminAcct/foo/parentDir/childDir’: File exists

I want to be able to consistently return a 'File exists' error.

background

I have --x group permission on the parent directory, /users/adminAcct/foo/parentDir. The child directory /users/adminAcct/foo/parentDir/childDir exists, and I have r-x group permissions on that directory. This originally started as a problem in python, but I was able to recreate it in the shell. I'm working on CentOS 6 workstations that are part of a cluster that has home directories mounted via NFS. The /users directory is in /etc/fstab, and is mounted with the options (rw,vers=3,hard,intr,addr=<some IP addr>). Kernel is Linux ws5.MyCompany.com 2.6.32-431.17.1.el6.x86_64 #1 SMP Wed May 7 23:32:49 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

context

For some time period after I access the child directory, mkdir will continue to return File exists errors. After some period, mkdir will once again return Permission denied errors. This creates intermittent problems starting and running airflow CeleryExecutor workers on these workstations, using /users/adminAcct/foo/parentDir/childDir as AIRFLOW_HOME. I haven't been able to predict when these problems will re-appear.

  • 1
    Is the NFS server running on Windows? If not, what is it, please – roaima Aug 22 '17 at 23:00
  • Can't tell exactly, from nmap it looks like it's using a BSD version of apache for https so probably not windows, at least – hhoke1 Aug 23 '17 at 0:51
  • Can you ask the sysadmins? – roaima Aug 23 '17 at 5:55
  • 2
    Is /users/adminAcct/foo/parentDir/childDir mounted as an auto-mount directory? Is the difference in error messages correlated with the mount status of the filesystem? Do you get Permission denied errors when the filesystem isn't mounted, and File exists errors when it is mounted? – Andrew Henle Aug 23 '17 at 9:34
  • The auto-mount idea seems like a good one. I was also able to recreate this behavior with a directory I know to be under automount. At first, /net/logs didn't show up in the results to mount and mkdir gave Permission denied. After I ran ls /net/logs/$targetDir, /net/logs showed up in the results to mount, and mkdir gave File exists. One hitch is that /etc/auto.master only contains /misc, /net. And /users/ is in /etc/fstab, and is consistently shown in the output of mount. – hhoke1 Aug 23 '17 at 15:24
1

The NFS client caches attributes, which have a wall clock based timeout.

This improves performance but gives 'slightly incorrect' results in some scenarios. The standard UNIX garantuees about file existance etc. are not always met by nature of the way that the client works.

One way to alleviate this is to disable client side caching (noac), or just directory entry caching (lookupcache=none) -- see https://linux.die.net/man/5/nfs

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.