1

I want to run a python script many times with various arguments. To do that, I've written the following bash script:

requests=(25 50 75 100)
factors=(3 6)
graphsizes=(25 50 75)

for request in "${requests[@]}"; do
    for factor in "${factors[@]}"; do
        for size in "${graphsizes[@]}"; do
            echo "Now Running: n = ${request}, factor = ${factor}, size = ${size}" >> nohup.out;
            echo nohup python3 -u main.py "$request" 50 "$factor" "$size" > ${request}_${factor}_${size}.log &
            echo "Done Running: n = ${request}, factor = ${factor}, size = ${size}" >> nohup.out;
        done
    done
done

I added ; at the end of the first and last echo because I do not want them to be run in parallel. In fact I want every call to the python script main.py to be run sequentially not in parallel, since the script itself is already parallelized and don't want any race conditions.

I know normally we use a ; to make jobs run sequentially, but if I do that after the & in the nohup line, I get the error

syntax error near unexpected token `;'

How do I make each iteration of the loop run sequentially?

2 Answers 2

2

The way you have written the script, on multiple lines, the ';' is implied by the linebreak. You can remove the ';' at the end of each of the three lines inside the innermost for loop and the script will run sequentially. You only need the ';' if you move things onto the same line when there should be a break between them, such as when you're putting the do on the same line as your for specification. Note the alternate placement of the do in the rewritten script below:

for request in "${requests[@]}" 
do
  for factor in "${factors[@]}"
  do
    for size in "${graphsizes[@]}"
    do
        echo "Now Running: n = ${request}, factor = ${factor}, size = ${size}" >> nohup.out
        echo nohup python3 -u main.py "$request" 50 "$factor" "$size" > ${request}_${factor}_${size}.log 
        echo "Done Running: n = ${request}, factor = ${factor}, size = ${size}" >> nohup.out
    done
  done
done
2
  • 1
    I'll note here that John has also removed the & after your invocation of Python. That will leave Python in the foreground so bash will wait until the script has completed and Python returns before it goes on to the next line. The & detaches the Python instance, so the bash script will proceed before main.py is done, which would seem to be not what you want.
    – tsc_chazz
    Nov 5, 2021 at 18:29
  • @tsc_chazz thank you!! That was exactly what I needed
    – a6623
    Nov 6, 2021 at 19:16
0

When then number of combinations gets bigger, you will overwhelm your computer.

This is where GNU Parallel comes in handy:

nohup parallel -j100% --header : python3 -u main.py {request} 50 {factor} {size} '>' {request}_{factor}_{size}.log \
   ::: request 25 50 75 100 \
   ::: factor 3 6 \
   ::: size 25 50 75

Change -j100% if you do not want run one job per CPU core.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.