2

I'm using GNU parallel to execute computation heavy transformations on remote hosts that are more powerful than the one requesting the work.

I am using a bash script located in /usr/local/lib/myscript, and I'd like to use it on the remote hosts. However, I don't want to require the hosts to have this script, so I'm sending it along the work with --basefile.

If I supply the filename only, all goes well. But if I supply the full path to a basefile, it doesn't work.

To illustrate:

$ cat /tmp/common.sh
#!/usr/bin/env bash
echo "Hello world! from $(hostname)"

$ cd /tmp
$ parallel --nonall -S 2/user@remote-host --basefile common.sh --cleanup bash common.sh
Hello world! from remote-host

$ parallel --nonall -S 2/user@remote-host --basefile     /tmp/common.sh --cleanup "bash {}"
could not make way for new symlink: tmp
rsync error: some files could not be transferred (code 23) at /BuildRoot/Library/Caches/com.apple.xbs/Sources/rsync/rsync-47/rsync/main.c(992) [sender=2.6.9]
cannot delete non-empty directory: tmp

Is there a way to use --basefile with files outside of the current path? Or should I use a second --trc and take advantage of the {1} and {2} substitution patterns to emulate --basefile?

Improve this question
1
  • Use /./ in the path.
    – Ole Tange
    Nov 28, 2016 at 8:36

1 Answer 1

Reset to default
2

Transferring of files (i.e. --basefile, --transfer-file, --transfer, --return) in GNU Parallel uses rsync's /./ magic. So when you ask to transfer /tmp/common.sh it will be put in /tmp/common.sh on the remote side. But if you ask to transfer /tmp/./common.sh it will transfer /tmp/common.sh to $(pwd)/..

In other words:

(local file) => (remote file)
dir/file => ./dir/file
/tmp/sub/dir/file => /tmp/sub/dir/file
/tmp/sub/./dir/file => ./dir/file

The error you see is probably due to /tmp being a symlink on the remote system.

If you use --workdir ... you will have new workdir for each job. As --basefile only copies the file before the first job is run, this will not work.

Instead you can transfer common.sh for each job:

parallel --wd ... --tf {1} -S server 'pwd ; ls; echo {2}' ::: /tmp/./common.sh ::: foo bar
Improve this answer
3
  • I don't think I understand how this works. If I observe the remote system where the command is executed, I can see the basefile transferred for the first job in ~/.parallel/tmp/host-1234/. But for the second and third jobs, only the files after ::: get transferred, the basefile isn't. And so the first job completes, cleans up, and the two other jobs fail because they don't have the basefile they need. I managed to sort out the /./ magic, but only the first job gets completed and retrieved.
    – springloaded
    Nov 28, 2016 at 22:26
  • Then please update the question: You forgot to mention you use ':::' and '--workdir' and not '--nonall'. It makes it harder to find a working solution for you if you leave that out.
    – Ole Tange
    Nov 29, 2016 at 7:59
  • My bad, I tried to simplify the command so readers don't get caught up in the details, I didn't understand workdir ... would have an impact.
    – springloaded
    Nov 29, 2016 at 15:29

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .