Waiting for any process to finish in bash script

Question

I am trying to check how many active processes are running in a bash script. The idea is to keep x processes running and when one finished then the next process is started.

For testing purposes I set up this script:

find $HOME/Downloads -name "dummy" &
find $HOME/Downloads -name "dummy" &
find $HOME/Downloads -name "dummy" &

while true
do
    pids=()
    while read pid; do
        echo "PID: $pid"
        pids+=("$pid")
    done < <(jobs -p)
    
    jobs -p

    echo "Active processes: ${#pids[@]}"
    if [ ${#pids[@]} -lt 2 ]; then
        break
    fi

    echo "Process(es) still running... ${pids[@]}"
    sleep 1
done

But this does not work because jobs -p continues to return the job ids even when the processes have finished.

The following example shows the problem in detail:

#!/bin/bash

find $HOME/Downloads -name "dummy" &
find $HOME/Downloads -name "dummy" &
find $HOME/Downloads -name "dummy" &

while true
do
    jobs -p # continues to print all 3 jobs
    sleep 1
done

How can I get the active jobs in the while loop?

Regards,

Answer 1

While wait -n (per comment @icarus) works in this particular situation, it should be noted that$! contains the PID of the last process that is started. So you could test on that as well:

#!/bin/bash

find $HOME/Downloads -name "dummy" &
p1=$!
find $HOME/Downloads -name "dummy" &
p2=$!
find $HOME/Downloads -name "dummy" &
p3=$!
while true
do
    if ps $p1 > /dev/null ; then
        echo -n "p1 runs "
    else 
        echo -n "p1 ended"
    fi
    if ps $p2 > /dev/null ; then
        echo -n "p2 runs "
    else 
        echo -n "p2 ended"
    fi
    if ps $p1 > /dev/null ; then
        echo -n "p3 runs "
    else 
        echo -n "p3 ended"
    fi
    echo ''
    sleep 1
done

But parallel is a better option.

Answer 2

The problem with the script is that there is nothing in it which is going to call one of the wait system calls. Generally until something calls wait the kernel is going to keep an entry for the process as this is where the return code of the child process is stored. If a parent process ends before a child process the child process is reparented, usually to PID 1. Once the system is booted, PID 1 often is programmed to enter a loop just calling wait to collect these processes exit value.

Rewriting the test script to call the shell builtin function wait we get

pids=()
find $HOME/Downloads -name "dummy" &
pids+=( $! )
find $HOME/Downloads -name "dummy" &
pids+=( $! )
find $HOME/Downloads -name "dummy" &
pids+=( $! )

echo "Initial active processes: ${#pids[@]}"
for ((i=${#pids[@]}; i>1; i--)) ; do 
do
    wait -n # Wait for  one process to exit
    echo "A process exited with RC=$?"
    # Note that -n is a bash extension, not in POSIX
    # if we have bash 5.1 then we can use "wait -np EX" to find which
    # job has finished, the value is put in $EX. Then we can remove the
    # value from the pids array. 

    echo "Still outstanding $(jobs -p)"
    sleep 1
done

Answer 3

I am trying to check how many active processes are running in a bash script. The idea is to keep x processes running and when one finished then the next process is started.

With GNU Parallel it looks like this:

parallel -j3 find $HOME/Downloads -name {} ::: "name1" "name2" "name3"

If you need to run different programs (and not just different arguments):

parallel -j3 ::: "find $HOME/Downloads -name name1" "find $HOME/Downloads -name name2"  "find $HOME/Downloads -name name3" 

If the commands are more complex, use bash functions:

find1() {
  find $HOME/Downloads -name "name1"
}
find2() {
  find $HOME/Downloads -name "name2"
}
find3() {
  find $HOME/Downloads -name "name3"
}
export -f find1 find2 find3
parallel -j3 ::: find1 find2 find3

If you leave out -j3 it will run one job per CPU core.