2

I pass the variables from a script to the main script in a parallel command like

./script1 | parallel -u --jobs 3 "./script2 {}"

How can I pass the job number as the second argument of ./script2?

something like

./script1 | parallel -u --jobs 3 "./script2 {} {}" ::: {1..3}

but the first {} should come from ./script1.

Note that I do not want the combination of arguments. Instead, the running jobs should be

./script2 var1 1
./script2 var2 2
./script2 var3 3
./script2 var4 1
./script2 var5 2
./script2 var6 3
./script2 var7 1

Imagine ./script2' writes to filesfile1.txt,files2.txt, andfile3.txtwhere the number is{1..3}` or the job number.

I want to make sure only one script is writing to its corresponding file. In other words, three parallel jobs write to three designated files.

2
  • Are you just looking for {#}?
    – meuh
    Commented Apr 24, 2020 at 16:11
  • @meuh yes, just the number of enough. {#} is perfect if the first {} reads the pipeline from the script.
    – Googlebot
    Commented Apr 24, 2020 at 16:31

2 Answers 2

1

You are most likely looking for {%} which is not the job number, but the job slot number.

LC_ALL=C seq 10 -0.1 1 | shuf |
  parallel --lb 'echo Job {} grabs {%}; sleep {}; echo Job {} releases {%}'

Notice how when a number is released it is grabbed by the next job. So no 2 jobs running in parallel will have the same {%} value.

This is covered in chapter 5 of GNU Parallel 2018 (http://www.lulu.com/shop/ole-tange/gnu-parallel-2018/paperback/product-23558902.html or https://doi.org/10.5281/zenodo.1146014).

2
  • I was trying to use this option with the numbering starting at 0. Is there an option to do this? (Background: I wanted to use the job id as variable for setting cuda visible devices).
    – Kam Sen
    Commented Jan 4 at 9:39
  • 1
    @KamSen parallel --rpl '{%0} 1 $_=$job->slot()-1' echo {%0} ::: a b c or parallel 'echo {=1 $_=$job->slot()-1=}' ::: a b c
    – Ole Tange
    Commented Jan 4 at 21:16
0

I found out that the solution is --link flag to aggregate the combinations.

parallel -u --jobs 3 --link ./script2 {1} {2} :::: <(./script1) ::: {1..3}

Then, {1} comes from the script, and {2} just matches periodically.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .