I have the following command in my Makefile
parallel \
--eta \
--bar \
--joblog mnist/embedder.joblog \
pipenv run python3 \
-m mnist.train_embedder \
--embedder_name {1} \
--embedder_dim {2} \
--embedder_lr {3} \
--embedder_epochs {4} \
:::: grid/embedder_name \
:::: grid/embedder_dim \
:::: grid/embedder_lr \
:::: grid/embedder_epochs
Each file contains something like
$ cat grid/embedder_name
ae
cnn
$ cat grid/embedder_dim
24
32
48
64
A dry run results in
$ parallel \
--dry-run \
--eta \
--joblog mnist/embedder.joblog \
pipenv run python3 \
-m mnist.train_embedder \
--embedder_name {1} \
--embedder_dim {2} \
--embedder_lr {3} \
--embedder_epochs {4} \
:::: grid/embedder_name \
:::: grid/embedder_dim \
:::: grid/embedder_lr \
:::: grid/embedder_epochs
Computers / CPU cores / Max jobs to run
1:local / 96 / 8
Computer:jobs running/jobs completed/%of started jobs/Average seconds to complete
ETA: 0s Left: 8 AVG: 0.00s local:8/0/100%/0.0s pipenv run python3 -m mnist.train_embedder --embedder_name ae --embedder_dim 24 --embedder_lr 0.001 --embedder_epochs 32
ETA: 0s Left: 7 AVG: 0.00s local:7/1/100%/0.0s pipenv run python3 -m mnist.train_embedder --embedder_name ae --embedder_dim 32 --embedder_lr 0.001 --embedder_epochs 32
ETA: 0s Left: 6 AVG: 0.00s local:6/2/100%/0.0s pipenv run python3 -m mnist.train_embedder --embedder_name ae --embedder_dim 48 --embedder_lr 0.001 --embedder_epochs 32
ETA: 0s Left: 5 AVG: 0.00s local:5/3/100%/0.0s pipenv run python3 -m mnist.train_embedder --embedder_name ae --embedder_dim 64 --embedder_lr 0.001 --embedder_epochs 32
ETA: 0s Left: 4 AVG: 0.00s local:4/4/100%/0.0s pipenv run python3 -m mnist.train_embedder --embedder_name cnn --embedder_dim 24 --embedder_lr 0.001 --embedder_epochs 32
ETA: 0s Left: 3 AVG: 0.00s local:3/5/100%/0.0s pipenv run python3 -m mnist.train_embedder --embedder_name cnn --embedder_dim 32 --embedder_lr 0.001 --embedder_epochs 32
ETA: 0s Left: 2 AVG: 0.00s local:2/6/100%/0.0s pipenv run python3 -m mnist.train_embedder --embedder_name cnn --embedder_dim 48 --embedder_lr 0.001 --embedder_epochs 32
ETA: 0s Left: 1 AVG: 0.00s local:1/7/100%/0.0s pipenv run python3 -m mnist.train_embedder --embedder_name cnn --embedder_dim 64 --embedder_lr 0.001 --embedder_epochs 32
ETA: 0s Left: 0 AVG: 0.00s local:0/8/100%/0.0s
The argument list is still growing, and if I want to add an argument between --embedder_name
and --embedder_dim
, I must edit --embedder_dim {2}
to --embedder_dim {3}
, --embedder_lr {3}
to --embedder_lr {4}
, and so on and so forth. This is tedious and error-prone.
Can I make the positional arguments into named arguments? I imagine something like the following
parallel \
--eta \
--bar \
--joblog mnist/embedder.joblog \
pipenv run python3 \
-m mnist.train_embedder \
--embedder_name {embedder_name} \
--embedder_dim {embedder_dim} \
--embedder_lr {embedder_lr} \
--embedder_epochs {embedder_epochs} \
:::: grid/embedder_name \
:::: grid/embedder_dim \
:::: grid/embedder_lr \
:::: grid/embedder_epochs
While this creates a lot of duplication of the names (the string embedder_name
appears thrice in the command!), at least it's more robust to accidentally incorrect order of arguments.
In case it's relevant
$ uname -a
Linux t1v-n-5d019513-w-0 5.13.0-1023-gcp #28~20.04.1-Ubuntu SMP Wed Mar 30 03:51:07 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
grid/embedder_<something>
and not the file in itself. If so, can you use the headers flag as follows?parallel --header : pipenv run python3 -m mnist.train_embedder --embedder_name {name} --embedder_dim {dim} --embedder_lr {lr} --embedder_epochs {epochs} ::: name grid/embedder_name ::: dim grid/embedder_dim ::: lr grid/embedder_lr ::: epochs grid/embedder_epochs
--header
, but this does not appear to work. I have edited the question to clarify my intention.--header
only works with positional replacement strings. In fact, I don't think GNU parallel supports named replacement strings in general :(