2

I have a python program that I run it via command line (Mac OSX) as:

python -W ignore Experiment.py --iterations 10

The file Experiment.py should be run multiple times using different --iterations values. I do that manually one after another, so when one run is finished, I run the second one with different --iterations, and so on. However, I cannot always set near to my laptop to run all of them so I am wondering if there is a way using shell script where I can state all runs together and then the shell script executes them one after another (Not parallel, just sequentially as I would have done it by my self)? Something like:

python -W ignore Experiment.py --iterations 10
python -W ignore Experiment.py --iterations 100
python -W ignore Experiment.py --iterations 1000
python -W ignore Experiment.py --iterations 10000
python -W ignore Experiment.py --iterations 100000

Edit: What if I have multiple arguments --X --Y --Z?

3

You can use a for loop:

for iteration in 10 100 1000 10000 100000; do
    python -W ignore Experiment.py --iteration "${iteration}"
done

If you have multiple parameters, and you want all the various permutations of all parameters, as @Fox noted in a comment below, you can use nested loops. Suppose, for example, you had a --name parameter whose values could be n1, n2, and n3, then you could do:

for iteration in 10 100 1000 10000 100000; do
    for name in n1 n2 n3; do
         python -W -ignore Experiment.py --iteration "${iteration}" --name "${name}"
    done
done

You could put that in a file, for example runExperiment.sh and include this as the first line: #!/bin/bash. You could then run the script using either:

bash runExperimen.sh

Or, you could make the script executable, then run it:

chmod +x runExperiment.sh
./runExperiment.sh

If you're interested in some results before others, that'll guide how you structure the loops. In my example above, the script will run:

... --iteration 10 --name n1
... --iteration 10 --name n2
... --iteration 10 --name n3
... --iteration 100 --name n1

So it runs all experiments for iteration 10 before moving on to the next iteration. If instead you wanted all experiments for name n1 before moving on to the next, you could make the name loop the "outer" loop.

for name in ...; do
    for iteration in ...; do
  • Oh Great! Thank you very very much. What if I have more than one parameters: --X --Y --Z, how do I make it ? – Katherine Jan 5 '18 at 14:59
  • Yes, if you want all permutations of all parameters, the nested loop will do what you're after. – Andy Dalton Jan 5 '18 at 15:05
  • @AndyDalton Thank you Andy. Appreciate your answer – Katherine Jan 5 '18 at 15:09
0

With GNU Parallel it looks like this:

parallel -j1 python -W ignore Experiment.py --iterations {} ::: 10 100 1000 10000 100000

Remove -j1 if you want to run one Experiment.py per CPU core in parallel.

To combine with multiple arguments, let us assume you have:

-s = S, M, L
--size = small, medium, large
--color =  red, green, blue

You want -s L to be linked to --size large but with all combinations of --color:

parallel echo s={1} size={2} color={3} ::: S M L :::+ small medium large ::: red green blue

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.