I am setting up a compute cluster on AWS. I'm using cfncluster which creates an NFS share on the master node which all compute nodes can access. Obviously this can become a bottleneck and I have seen it in my previous work with cfncluster. I previously addressed it by using a SLURM prolog which copied input files to the local node and updated job references to the NFS path to go to the local copy instead. Likewise, I redirected the output path of the job to a local path and added a line in the job script to copy the output file back to the NFS path after the job finishes.
I just learned about FS-Cache for NFS today. This seems like it could replace my input file copying, but only if I can make it copy the full file whenever a program asks for a part of a file on the NFS path. Perhaps copying the input files from NFS and immediately deleting them would keep the file cached?
The other thing I want is to cache the output file so that if the next job that runs on that node needs that output file, it doesn't have to copy it back from NFS. It seems that FS-Cache doesn't cache writes to NFS, so I'm not sure it can accomplish that. Please enlighten me if I'm wrong!
cat file > /dev/nullought to suffice).while true; do cat file > /dev/null; sleep 10; done