GNU parallel - two parameters from array as parameter

Question

I have a script that uses gnu parallel. I want to pass two parameters for each "iteration"

in serial run I have something like:

for (( i=0; i<=10; i++ ))
do
  a = tmp1[$i]
  b = tmp2[$i]
done

And I want to make this parallel as

func pf()
{
   a=$1
   b=$2
}
export -f pf
parallel --jobs 5 --linebuffer pf ::: <what to write here?>

Answer 1

Omitting your other parallel flags just to stay focused...

parallel --link pf ::: A B ::: C D

This will run your function first with a=A, b=C followed by a=B, b=D or

a=A b=C
a=B b=D

Without --link you get full combination like this:

a=A b=C
a=A b=D
a=B b=C
a=B b=D

---

Update: As Ole Tange metioned in a comment [since deleted - Ed.] there is another way to do this: use the :::+ operator. However, there is an important difference between the two alternatives if the number of arguments is not the same in each param position. An example will illustrate.

parallel --link pf ::: A B ::: C D E output:

a=A b=C
a=B b=D
a=A b=E

parallel pf ::: A B :::+ C D E output:

a=A b=C
a=B b=D

So --link will "wrap" such that all arguments are consumed while :::+ will ignore the extra argument. (In the general case I prefer --link since the alternative is in some sense silently ignoring input. YMMV.)

Answer 2

To simplify, I will assume bash and that the arrays are indexed from 1 rather than 0. It seems intuitive to want to do something like this:

parallel ... pf '$tmp1[{#}]' '$tmp2[{#}]' ::: $(seq 10)

where the two args to your pf function are part of the command, and we use the parallel notation {#} to stand for the job number (which is set to 1 up to 10 for 10 jobs. We simply use seq to get 10 arguments after the ::: to ensure we do 10 jobs. (The seq values are not used, and just happen to be the same as the job numbers.)

Unfortunately, this will not work as bash does not export array variables. But it can export functions, and the parallel man page shows the workaround using a simple import_array function to export/import a function of your choice, my_importer that will set the array variables of your choice:

declare -a tmp1 tmp2
for (( i=1; i<=10; i++ ))
do tmp1[$i]=x$i
   tmp2[$i]=y$i
done

import_array(){
    local func=$1; shift;
    export $func='() {
      '"$(for arr in $@; do
            declare -p $arr|sed '1s/declare -./&g/'
          done)"'
    }'
}
import_array my_importer tmp1 tmp2

We neeed only tell parallel to pass the my_importer function into the environment of the pf command, with option --env my_importer, and then run that function before running pf:

pf(){ a=$1; b=$2; echo "job a=$a b=$b"; }
export -f pf 

parallel -v --jobs 5 --linebuffer \
 --env my_importer 'my_importer;'  pf '${tmp1[{#}]}' '${tmp2[{#}]}' ::: $(seq 10)

The resulting output with -v is similar to

my_importer; pf ${tmp1[2]} ${tmp2[2]}
my_importer; pf ${tmp1[1]} ${tmp2[1]}
my_importer; pf ${tmp1[5]} ${tmp2[5]}
my_importer; pf ${tmp1[3]} ${tmp2[3]}
job a=x1 b=y1
my_importer; pf ${tmp1[6]} ${tmp2[6]}
job a=x2 b=y2
my_importer; pf ${tmp1[7]} ${tmp2[7]}
job a=x4 b=y4
...