8

Is this the correct way to start multiple sequential processings in the background?

for i in {1..10}; do
    for j in {1..10}; do
        run_command $i $j;
    done &
done;

All j should be processed after each other for a given i, but all i should be processed simultaneously.

  • What do you mean by "correct way"? – Eran Ben-Natan Oct 3 '19 at 10:17
  • Mainly that it works. I was missing this on stackoverflow. – Radio Controlled Oct 3 '19 at 17:41
  • 1
    :-) I'm asking because your code works fine for me. If by "correct way" you mean best practice, then I would put inner loop in a function or even different file so its more clear – Eran Ben-Natan Oct 6 '19 at 6:13
8

The outer loop that you have is basically

for i in {1..10}; do
    some_compound_command &
done

This would start ten concurrent instances of some_compound_command in the background. They will be started as fast as possible, but not quite "all at the same time" (i.e. if some_compound_command takes very little time, then the first may well finish before the last one starts).

The fact that some_compound_command happens to be a loop is not important. This means that the code that you show is correct in that iterations of the inner j-loop will be running sequentially, but all instances of the inner loop (one per iteration of the outer i-loop) would be started concurrently.

The only thing to keep in mind is that each background job will be running in a subshell. This means that changes made to the environment (e.g. modifications to values of shell variables, changes of current working directory with cd, etc.) in one instance of the inner loop will not be visible outside of that particular background job.

What you may want to add is a wait statement after your loop, just to wait for all background jobs to actually finish, at least before the script terminates:

for i in {1..10}; do
    for j in {1..10}; do
        run_command "$i" "$j"
    done &
done

wait
3

If you have GNU Parallel you would do:

parallel -j0 'for j in {1..10}; do run_command {} $j; done' ::: {1..10}

One of the benefits is that output from parallel running run_commands will not mix.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.