I want to run some simulations using a Python tool that I had made. The catch is that I would have to call it multiple times with different parameters/arguments and everything.

For now, I am using multiple for loops for the task, like:

for simSeed in 1 2 3 4 5
    for launchPower in 17.76 20.01 21.510 23.76
        python sim -a $simSeed -p $launchPower

In order for the simulations to run simultaneously, I append a & at the end of the line where I call the simulator.

python sim -a $simSeed -p $launchPower &

Using this method I am able to run multiple such seeds. However, since my computer has limited memory, I want to re-write the above script so that it launches the inner for loop parallelly and the outer for loop sequentially.

As an example, for simSeed = 1, I want 5 different processes to run with launchPower equal to 17.76 20.01 21.510 23.76. As soon as this part is complete, I want the script to run for simSeed = 2 and again 5 different parallel processes with launchPower equal to 17.76 20.01 21.510 23.76.

How can I achieve this task?


I want the outer loop to run sequentially and inner loop to run parallelly such that when the last parallel process of the inner loop finishes, the outer loop moves to the next iteration.

  • 1
    Welcome to the site. This looks like a job for GNU parallel. – AdminBee Mar 13 '20 at 10:54
  • How can I achieve this task using parallel? Isn't it similar to using the & command? (I don't have much XP in using parallel) – MaJoR21 Mar 13 '20 at 10:57
  • 1
    add wait between done and done – frostschutz Mar 13 '20 at 11:01
  • 1
    Why would launchPower be 1,2,3,4,5 ? Your loop is using 17.76 20.01 21.510 23.76 instead. Where does the 1,2,3,4,5 come from? – terdon Mar 13 '20 at 11:07
  • @terdon ach... I mixed up the values... you are correct... Lemme fix the question – MaJoR21 Mar 13 '20 at 11:11

GNU parallel has several options to limit resource usage when starting jobs in parallel.

The basic usage for two nested loops would be

parallel python sim -a {1} -p {2} ::: 1 2 3 4 5 ::: 17.76 20.01 21.510 23.76

If you want to launch at most 5 jobs at the same time, e.g., you could say

parallel -j5 python <etc.>

Alternatively, you can use the --memfree option to start new jobs only when enough memory is free, e.g. at least 256 MByte

parallel --memfree 256M python <etc.>

Note that the last option will kill the most recently started job if the memory falls below 50% of the "reserve" value stated (but it will be re-qeued for catch-up automatically).

  • This is exactly what I was looking for... If I have multiple such jobs (queued one after the other) then using the -j option it sequentially executes the first j jobs, right? – MaJoR21 Mar 13 '20 at 11:17
  • 1
    Yes, that is correct. Once the first of these j jobs has finished, it will start the first of the "remaining" jobs in the loop order, (i.e. Job j+1) and so on. – AdminBee Mar 13 '20 at 11:17
  • That said, note that parellel only guarantees the order in which jobs are started, not necessarily the order in which they are completed. For example with -j5, if job 2 takes 300s and job 3 only 20s, job 6 will be started before job 2 is complete. If job 6 also only takes 20s, its results will be "on disk" before those of job 2, which can lead to confusion. Although, now that I think about it ... it should be obvious, right ;)? – AdminBee Mar 13 '20 at 11:25
  • 1
    @MaJoR21 just send it to the background: parallel -j5 python <etc.> & – terdon Mar 13 '20 at 11:51
  • 2
    @MaJoR21 sounds like you really want to look into GNU screen or tmux as well. – terdon Mar 13 '20 at 11:56

As mentioned in the comments this is exactly what GNU parallel is for:

for simSeed in 1 2 3 4 5
    ## Launch 5 instances in parallel 
    parallel -j5 python sim -a $simSeed -p {} ::: 17.76 20.01 21.510 23.76

You could store the respective process IDs and wait for them to finish:

for simSeed in {1..5}; do
  for launchPower in 17.76 20.01 21.510 23.76; do
    python sim -a $simSeed -p $launchPower &
  wait ${pids[@]}

pids is an array of process IDs of the background jobs of the inner loop. With the ${pids[@]} all elements of the array are passed to the wait command.

  • Thanks for your answer... Since your solution is quite similar to my current setup, I will try this first... However, parallel does seem easy to use. – MaJoR21 Mar 13 '20 at 11:18
  • 2
    Note that you don't actually need to save the PIDs, wait will wait for any child processes by default. So all you need is for simSeed in {1..5}; do for launchPower in ...; do ...; done; wait; done; – terdon Mar 13 '20 at 11:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.