GNU Parallel with named arguments

Question

I have the following command in my Makefile

parallel \
        --eta \
        --bar \
        --joblog mnist/embedder.joblog \
        pipenv run python3 \
        -m mnist.train_embedder \
        --embedder_name {1} \
        --embedder_dim {2} \
        --embedder_lr {3} \
        --embedder_epochs {4} \
        :::: grid/embedder_name \
        :::: grid/embedder_dim \
        :::: grid/embedder_lr \
        :::: grid/embedder_epochs

Each file contains something like

$ cat grid/embedder_name
ae
cnn
$ cat grid/embedder_dim
24
32
48
64

A dry run results in

$ parallel \
        --dry-run \
        --eta \
        --joblog mnist/embedder.joblog \
        pipenv run python3 \
        -m mnist.train_embedder \
        --embedder_name {1} \
        --embedder_dim {2} \
        --embedder_lr {3} \
        --embedder_epochs {4} \
        :::: grid/embedder_name \
        :::: grid/embedder_dim \
        :::: grid/embedder_lr \
        :::: grid/embedder_epochs

Computers / CPU cores / Max jobs to run
1:local / 96 / 8

Computer:jobs running/jobs completed/%of started jobs/Average seconds to complete
ETA: 0s Left: 8 AVG: 0.00s  local:8/0/100%/0.0s pipenv run python3 -m mnist.train_embedder --embedder_name ae --embedder_dim 24 --embedder_lr 0.001 --embedder_epochs 32
ETA: 0s Left: 7 AVG: 0.00s  local:7/1/100%/0.0s pipenv run python3 -m mnist.train_embedder --embedder_name ae --embedder_dim 32 --embedder_lr 0.001 --embedder_epochs 32
ETA: 0s Left: 6 AVG: 0.00s  local:6/2/100%/0.0s pipenv run python3 -m mnist.train_embedder --embedder_name ae --embedder_dim 48 --embedder_lr 0.001 --embedder_epochs 32
ETA: 0s Left: 5 AVG: 0.00s  local:5/3/100%/0.0s pipenv run python3 -m mnist.train_embedder --embedder_name ae --embedder_dim 64 --embedder_lr 0.001 --embedder_epochs 32
ETA: 0s Left: 4 AVG: 0.00s  local:4/4/100%/0.0s pipenv run python3 -m mnist.train_embedder --embedder_name cnn --embedder_dim 24 --embedder_lr 0.001 --embedder_epochs 32
ETA: 0s Left: 3 AVG: 0.00s  local:3/5/100%/0.0s pipenv run python3 -m mnist.train_embedder --embedder_name cnn --embedder_dim 32 --embedder_lr 0.001 --embedder_epochs 32
ETA: 0s Left: 2 AVG: 0.00s  local:2/6/100%/0.0s pipenv run python3 -m mnist.train_embedder --embedder_name cnn --embedder_dim 48 --embedder_lr 0.001 --embedder_epochs 32
ETA: 0s Left: 1 AVG: 0.00s  local:1/7/100%/0.0s pipenv run python3 -m mnist.train_embedder --embedder_name cnn --embedder_dim 64 --embedder_lr 0.001 --embedder_epochs 32
ETA: 0s Left: 0 AVG: 0.00s  local:0/8/100%/0.0s

The argument list is still growing, and if I want to add an argument between --embedder_name and --embedder_dim, I must edit --embedder_dim {2} to --embedder_dim {3}, --embedder_lr {3} to --embedder_lr {4}, and so on and so forth. This is tedious and error-prone.

Can I make the positional arguments into named arguments? I imagine something like the following

parallel \
        --eta \
        --bar \
        --joblog mnist/embedder.joblog \
        pipenv run python3 \
        -m mnist.train_embedder \
        --embedder_name {embedder_name} \
        --embedder_dim {embedder_dim} \
        --embedder_lr {embedder_lr} \
        --embedder_epochs {embedder_epochs} \
        :::: grid/embedder_name \
        :::: grid/embedder_dim \
        :::: grid/embedder_lr \
        :::: grid/embedder_epochs

While this creates a lot of duplication of the names (the string embedder_name appears thrice in the command!), at least it's more robust to accidentally incorrect order of arguments.

In case it's relevant

$ uname -a
Linux t1v-n-5d019513-w-0 5.13.0-1023-gcp #28~20.04.1-Ubuntu SMP Wed Mar 30 03:51:07 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Answer 1

--header : does almost what you want.

It is made for CSV-files where the first line is a header.

So you need to prepend 'grid/embedder_name' with 'name':

$ cat grid/embedder_name
name
ae
cnn
$ cat grid/embedder_dim
dim
24
32
48
64

Also you do not need :::: between every file. A single one is enough (but if you find it easier to read, keep them):

parallel --header : echo {name} {dim} :::: grid/embedder_dim  grid/embedder_name

(@r_31415 shows a non-working example where the name is put on the command line. You do this if you use ::: - not :::: Example: parallel --header : echo {foo} ::: foo 1 2 3)

Edit:

From version 20220822 you can do:

parallel --header 0 echo {grid/embedder_name} {grid/embedder_dim} :::: grid/embedder_dim  grid/embedder_name
parallel --header 0 echo {embedder_name} {embedder_dim} :::: embedder_dim  embedder_name

For this the files should not have a header in the first line.

Answer 2

Have thought about wrapping it in a script/function? E.g.

#!/bin/bash
genarglist() {
for ((i=1;i<=$#;i++)) ; do
  echo -n " --embedder_${!i} {$i}"
done
for ((i=1;i<=$#;i++)) ; do
  echo -n " :::: grid/embedder_${!i}"
done
}

parallel --eta --bar --joblog mnist/embedder.joblog \
    pipenv run python3 -m mnist.train_embedder \
    $(genarglist $*)

and then just supply the name, dim ... as arguments.