Running python script many times in a bash file

Question

I want to run a python script many times with various arguments. To do that, I've written the following bash script:

requests=(25 50 75 100)
factors=(3 6)
graphsizes=(25 50 75)

for request in "${requests[@]}"; do
    for factor in "${factors[@]}"; do
        for size in "${graphsizes[@]}"; do
            echo "Now Running: n = ${request}, factor = ${factor}, size = ${size}" >> nohup.out;
            echo nohup python3 -u main.py "$request" 50 "$factor" "$size" > ${request}_${factor}_${size}.log &
            echo "Done Running: n = ${request}, factor = ${factor}, size = ${size}" >> nohup.out;
        done
    done
done

I added ; at the end of the first and last echo because I do not want them to be run in parallel. In fact I want every call to the python script main.py to be run sequentially not in parallel, since the script itself is already parallelized and don't want any race conditions.

I know normally we use a ; to make jobs run sequentially, but if I do that after the & in the nohup line, I get the error

syntax error near unexpected token `;'

How do I make each iteration of the loop run sequentially?

Answer 1

The way you have written the script, on multiple lines, the ';' is implied by the linebreak. You can remove the ';' at the end of each of the three lines inside the innermost for loop and the script will run sequentially. You only need the ';' if you move things onto the same line when there should be a break between them, such as when you're putting the do on the same line as your for specification. Note the alternate placement of the do in the rewritten script below:

for request in "${requests[@]}" 
do
  for factor in "${factors[@]}"
  do
    for size in "${graphsizes[@]}"
    do
        echo "Now Running: n = ${request}, factor = ${factor}, size = ${size}" >> nohup.out
        echo nohup python3 -u main.py "$request" 50 "$factor" "$size" > ${request}_${factor}_${size}.log 
        echo "Done Running: n = ${request}, factor = ${factor}, size = ${size}" >> nohup.out
    done
  done
done

Answer 2

When then number of combinations gets bigger, you will overwhelm your computer.

This is where GNU Parallel comes in handy:

nohup parallel -j100% --header : python3 -u main.py {request} 50 {factor} {size} '>' {request}_{factor}_{size}.log \
   ::: request 25 50 75 100 \
   ::: factor 3 6 \
   ::: size 25 50 75

Change -j100% if you do not want run one job per CPU core.