I am trying to use a Python script to read/write from an external disk in linux, which works fine. The problem comes when I use a job scheduler to execute the Python program.

I am running all this on a Rocks Cluster (the linux version is CentOS 6.5) and the job scheduler is Sun Grid Engine (SGE). I've mounted the external disk such that all users have read and write permissions. The line from fstab for mounting the disk is:

/dev/sdb1 /mnt/drive   ntfs-3g    auto,users,permissions   0   0

Let me give you a simple example, which fails. The following python program (simple_write.py):

#!/usr/bin/env python2.7

f = open('/mnt/drive/sean/test.txt', 'w')
f.write('Writing in the file')

runs fine if executed in a terminal like this:

python2.7 /home/sean/simple_write.py

If I then create a little bash script (job_submit.sh) for submitting to the scheduler (SGE) like this:

#$ -cwd
#$ -j y
#$ -S /bin/bash
python2.7 /home/sean/simple_write.py

and submit from a terminal:

qsub job_submit.sh

Then I get the following (Python) error:

IOError: [Errno 2] No such file or directory: '/mnt/drive/sean/test.txt'

The problem can be resolved by (a) running simple_write.py directly from the terminal or (b) changing the write location in simple_write.py to somewhere on the local disk, rather than the external disk (a similar error occurs for reading a file/directory rather than writing). It therefore seems like it must be some issue with the scheduler not playing nicely with the mounted drive, but I'm at a loss at what to try next. Any help would be greatly appreciated.

  • 1
    How many systems are in this cluster? That is, is the SGE job being run on some other system that cannot see that drive on the master node (NFS may then be necessary to share that directory). – thrig Sep 15 '15 at 17:15
  • To investigate: try submitting a script that does ls -la /mnt/drive/sean. – G-Man Sep 15 '15 at 17:27
  • @thrig that is almost certainly the issue. Let me go an investigate and get back to you. – Sean Elvidge Sep 16 '15 at 7:28
  • @thrig you were exactly right. I shared the directory using NFS and now all works fine. If you want to write your comment as an answer then I can accept it? – Sean Elvidge Sep 16 '15 at 14:03

Cluster job schedulers (e.g. Sun Grid Engine, HTCondor, SLURM, and so forth) may run jobs on some other system than the master or job submission host; filesystems only exposed on the master or job submission host thus may be unavailable to job-running systems in the cluster. The job scheduler may have logs or flags to indicate what system a job is running on, or available filesystems can be inspected by submitting a job that performs ls or df on whatever system it is run on (though, beware, such jobs may block on I/O if there is an unavailable NFS hard mount, in which case the job would be stuck according to the cluster software) to determine whether a directory or what filesystems are available. If the filesystem is unavailable, it will need to be made available via NFS (or some other network file system). Another option may be to have the job scheduler software copy the necessary files to the host the job will be run on, though how to do this will vary depending on job scheduler used (and, presumably, any output files would need to be copied elsewhere once the job completes).

Whether to copy or depend on a network filesystem depends on the quantity of data, a small amount of data could easily be copied, while a large repository of data the jobs will rummage through would favor using a network filesystem. Also note that a large number of clients can easily overload a NFS server; output (especially temporary output) may need to be written to a local filesystem on the job-running system, and only the results written to the NFS server.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.