Start 100 process at a time in bash script

Question

In bash script, I have a program like this

for i in {1..1000}
do
   foo i
done

Where I call the function foo 1000 times with parameter i

If I want to make it run in multi-process, but not all at once, what should I do?

So if I have

for i in {1..1000}
do
   foo i &
done

It would start all 1000 processes at once, which is not what I want.

Is there a way to make sure that there is always 100 process running? If some processes are finished, start some new ones, until all 1000 iterations are done. Alternatively, I could wait till all 100 are finished and run another 100.

Answer 1

With zsh instead of bash:

autoload -Uz zargs
zargs -P100 -I{} -- {1..1000} -- foo {}

But if you have GNU xargs, you can also do (in zsh, ksh93 or bash):

xargs -I{} -P100 -a <(echo {1..1000}) foo {}

foo has to be a standalone command though. It won't work with a shell function or builtin.

Note that zsh's zargs runs one batch after the other: starts 100 jobs, waits for all of them to return and only then starts that next batch of 100. While GNU xargs, will try to keep up to 100 running: start 100 and then start another one every time one finishes.

To get that xargs behaviour, in zsh you could start and manage your pool of jobs in a SIGCHLD trap which is triggered whenever a background process returns:

(
  todo=( {1..1000} ) max=100

  TRAPCHLD() {
    while (( $#jobstates < max && $#todo )); do
      foo $todo[1] & shift 1 todo
    done
  }

  : start &
  while (( $#todo )) wait
)

Here, we need to run it in a subshell to get a fresh job list though. SIGCHLDs are blocked while the TRAPCHLD trap is being run so the trap should not re-enter itself which should avoid race conditions or the need to protect from concurrent access to the $todo list.

Answer 2

If you are OK to run as groups, nest a loop:

#! /bin/bash

date '+%T.%N'
for j in {1..3}; do
    for k in {1..3}; do
        (( ++i ))
        ( sleep 2.0 && printf 'Foo %d\n' $i ) &
    done
    wait
    date '+%T.%N'
    printf 'Batch %d ends\n' $j 
done
date '+%T.%N'

Results, showing time overlaps:

$ ./aBatch
19:55:17.078476713
Foo 1
Foo 2
Foo 3
19:55:19.094302514
Batch 1 ends
Foo 4
Foo 6
Foo 5
19:55:21.114530543
Batch 2 ends
Foo 7
Foo 9
Foo 8
19:55:23.132184671
Batch 3 ends
19:55:23.135792952
$ 

This is the same thing in GNU parallel. This has the advantage that, if the executions run for different times, parallel will start further processes without waiting for others in the batch.

#! /bin/bash
#.. The script ./aFoo

    sleep 2 && printf 'Foo %d\n' $1

The command:

$ date '+%T.%N'; parallel -j 3 ./aFoo -- {1..9}; date '+%T.%N'
20:11:44.446042653
Foo 3
Foo 1
Foo 2
Foo 4
Foo 5
Foo 6
Foo 7
Foo 8
Foo 9
20:11:50.503324162
$ 

Answer 3

here's a simple way to do them in chunks of 100, (bash)

for i in {1..1000}
do
   foo "$i" &
   (( i % 100 )) || wait
done
wait

It assumes that there are no other background tasks running in the same shell, just starting 100 tasks and then waitning for them all to comolete before starting another 100, and finally waiting for any remainder to complete. (with 1000 and 100 there is no remainder, but in other cases there may be)

if your loop variable is not numeric you can use ++n instead of i in the expression

n=0
for i in *
do
   foo "$i" &
   (( ++n % 100 )) || wait
done
wait

Answer 4

#!/bin/bash

jobs_to_run_num=10
simult_jobs_num=3
have_runned_jobs_cntr=0
check_interval=0.1

while ((have_runned_jobs_cntr < jobs_to_run_num)); do 
    cur_jobs_num=$(wc -l < <(jobs -r))

    if ((cur_jobs_num < simult_jobs_num)); then
        ./random_time_script.sh &
        echo -e "cur_jobs_num\t$((cur_jobs_num + 1))"
        ((have_runned_jobs_cntr++))
    # sleep is needed to reduce the frequency of while loop
    # otherwise it itself will eat a lot of processor time
    # by restlessly checking
    else
        sleep "$check_interval"
    fi  
done

The better way - by using wait -n. No need for checking jobs number every iteration and usage of sleep command.

jobs_to_run_num=10
simult_jobs_num=3

while ((have_runned_jobs_cntr < jobs_to_run_num)); do
    if (( i++ >= simult_jobs_num )); then
        wait -n   # wait for any job to complete. New in 4.3
    fi
    ./random_time_script.sh &
    ((have_runned_jobs_cntr++))

    # For demonstration
    cur_jobs_num=$(wc -l < <(jobs -r))
    echo -e "cur_jobs_num\t${cur_jobs_num}"
done 

Idea from here - I want to process a bunch of files in parallel, and when one finishes, I want to start the next. And I want to make sure there are exactly 5 jobs running at a time.

Testing

$ ./test_simult_jobs.sh 
cur_jobs_num    1
cur_jobs_num    2
cur_jobs_num    3
cur_jobs_num    3
cur_jobs_num    3
cur_jobs_num    3
cur_jobs_num    3
cur_jobs_num    3
cur_jobs_num    3
cur_jobs_num    3

Answer 5

GNU Parallel is build for exactly this kind of situation:

parallel foo ::: {1..1000}

This will run one foo n per CPU thread (where n = 1..1000) until all 1000 jobs are run. When one finishes, another one is started.

parallel -j100 foo ::: {1..1000}

This will run 100 foo n (where n = 1..1000) until all 1000 jobs are run.

GNU Parallel will serialize the output, so if two foos print at the same time, the output will not be garbled.

GNU Parallel has many other features that can make parallelization easier: Spend 20 minutes on reading chapter 1+2 https://zenodo.org/record/1146014 Your command line will thank you for it.

Answer 6

To avoid having to continually poll for jobs finishing, you can spawn the 100 subshells and have them read $i from stdin until they have all been processed. Instead of doing "for i in" with echo, here I'm using seq to list the 1000 numbers:

LOCKFILE=`pwd` # choose a file that is guaranteed to exist and be readable
seq 1000 | for worker in {1..100}; do # spawn worker subshells
    ( while flock -x 356 && read i; do foo $i; done ) &
done 356<"$LOCKFILE" |cat

• When one "foo" finishes, another is immediately started with the next available "i" from the list.
• flock protects against more than one "read i" happening at a time. `pwd` is convenient for LOCKFILE, but assumes that the cwd is readable. If this might not be true in your case, alternatively create a new file and remove it again afterwards. 356 is a random number between 201 and 1000
• The "|cat" waits for all the spawned subshells to finish - it can be dropped if you don't care, e.g. you're checking in top anyway, or you are running a few of these tasks concurrently and are just limiting the impact of each one.