3

I am setting up a compute cluster on AWS. I'm using cfncluster which creates an NFS share on the master node which all compute nodes can access. Obviously this can become a bottleneck and I have seen it in my previous work with cfncluster. I previously addressed it by using a SLURM prolog which copied input files to the local node and updated job references to the NFS path to go to the local copy instead. Likewise, I redirected the output path of the job to a local path and added a line in the job script to copy the output file back to the NFS path after the job finishes.

I just learned about FS-Cache for NFS today. This seems like it could replace my input file copying, but only if I can make it copy the full file whenever a program asks for a part of a file on the NFS path. Perhaps copying the input files from NFS and immediately deleting them would keep the file cached?

The other thing I want is to cache the output file so that if the next job that runs on that node needs that output file, it doesn't have to copy it back from NFS. It seems that FS-Cache doesn't cache writes to NFS, so I'm not sure it can accomplish that. Please enlighten me if I'm wrong!

3
  • 1
    Curious about wanting to grab the full file on any access—wouldn't that increase the load on your NFS server? (BTW: to get the full file cached, cat file > /dev/null ought to suffice).
    – derobert
    Commented Aug 30, 2017 at 20:20
  • @derobert Or, depending on the cache size. filesystem activity, the size of and access to the file, and the time it is needed in the cache: while true; do cat file > /dev/null; sleep 10; done Commented Aug 30, 2017 at 20:34
  • 1
    Grabbing the whole file at once in a sequential copy will reduce network contention. These tools usually read some, do some work, read some more, do some more work, etc. So they're always asking for bits of the file over the network. Commented Aug 30, 2017 at 20:48

0

You must log in to answer this question.

Browse other questions tagged .