I'm using GNU parallel to execute computation heavy transformations on remote hosts that are more powerful than the one requesting the work.

I am using a bash script located in /usr/local/lib/myscript, and I'd like to use it on the remote hosts. However, I don't want to require the hosts to have this script, so I'm sending it along the work with --basefile.

If I supply the filename only, all goes well. But if I supply the full path to a basefile, it doesn't work.

To illustrate:

$ cat /tmp/common.sh
#!/usr/bin/env bash
echo "Hello world! from $(hostname)"

$ cd /tmp
$ parallel --nonall -S 2/user@remote-host --basefile common.sh --cleanup bash common.sh
Hello world! from remote-host

$ parallel --nonall -S 2/user@remote-host --basefile     /tmp/common.sh --cleanup "bash {}"
could not make way for new symlink: tmp
rsync error: some files could not be transferred (code 23) at /BuildRoot/Library/Caches/com.apple.xbs/Sources/rsync/rsync-47/rsync/main.c(992) [sender=2.6.9]
cannot delete non-empty directory: tmp

Is there a way to use --basefile with files outside of the current path? Or should I use a second --trc and take advantage of the {1} and {2} substitution patterns to emulate --basefile?

  • Use /./ in the path. – Ole Tange Nov 28 '16 at 8:36

Transferring of files (i.e. --basefile, --transfer-file, --transfer, --return) in GNU Parallel uses rsync's /./ magic. So when you ask to transfer /tmp/common.sh it will be put in /tmp/common.sh on the remote side. But if you ask to transfer /tmp/./common.sh it will transfer /tmp/common.sh to $(pwd)/..

In other words:

(local file) => (remote file)
dir/file => ./dir/file
/tmp/sub/dir/file => /tmp/sub/dir/file
/tmp/sub/./dir/file => ./dir/file

The error you see is probably due to /tmp being a symlink on the remote system.

If you use --workdir ... you will have new workdir for each job. As --basefile only copies the file before the first job is run, this will not work.

Instead you can transfer common.sh for each job:

parallel --wd ... --tf {1} -S server 'pwd ; ls; echo {2}' ::: /tmp/./common.sh ::: foo bar
| improve this answer | |
  • I don't think I understand how this works. If I observe the remote system where the command is executed, I can see the basefile transferred for the first job in ~/.parallel/tmp/host-1234/. But for the second and third jobs, only the files after ::: get transferred, the basefile isn't. And so the first job completes, cleans up, and the two other jobs fail because they don't have the basefile they need. I managed to sort out the /./ magic, but only the first job gets completed and retrieved. – springloaded Nov 28 '16 at 22:26
  • Then please update the question: You forgot to mention you use ':::' and '--workdir' and not '--nonall'. It makes it harder to find a working solution for you if you leave that out. – Ole Tange Nov 29 '16 at 7:59
  • My bad, I tried to simplify the command so readers don't get caught up in the details, I didn't understand workdir ... would have an impact. – springloaded Nov 29 '16 at 15:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.