Bash – listen for exit of background process

Question

I have this bash script to start some servers:

services=(
    account-service
    reminder-service
    activity-service
    socket-service
    chat-service
    web-app
)


for s in "${services[@]}"; do

 (
   set -e;
   cd "$s"
   git pull
   npm start || exit 1 # always fails
 ) &

 sleep 1;

done

wait;

Sometimes the git pull command will fail. But the failure is deep in the logs and not always obvious. How can I abort the whole script if one of the commands in the subshell exits with 1?

Answer 1

Hmm, maybe: change the wait statement to:

while true; do
    wait -n || exit 1          # if one of the background jobs failed, abort
    [[ "$(jobs)" ]] && break   # exit this loop if no more jobs
done

Completely untested. I'm not sure if jobs works as expected in a non-interactive shell.

Answer 2

There are a couple of issues here:

1. Getting the whole script to abort if a command in a subshell fails.
2. Raising the visibility of any errors that occur.

I’ll start with the second one.  The question contains the seed of the answer: errors are logged, but the content of the logs is not obvious to the user.  So log the errors (in an ad-hoc log file) and display them at the end:

services=(
    account-service
    reminder-service
    activity-service
    socket-service
    chat-service
    web-app
)
errfile=$(mktemp)

for s in "${services[@]}"; do
 (
   set -e;
   cd "$s"
   git pull  ||  {
        echo "$s" >> "$errfile"
        exit 1
   }
   echo "$cmd" | bash
 ) &
 sleep 1
done

wait

if [ -s "$errfile" ]
then
        echo "The following service(s) had errors:"
        cat "$errfile"
fi
rm -f "$errfile"

The simplest way to address issue #1 is to have the code in the subshell(s) check [ -s "$errfile" ] periodically, and abort itself if any problem arises.  A more ambitious approach would be to have the main (parent) process keep track of the PIDs of the children and send them signals.  I’m not sure how you would do that without wait -n.  (Maybe you could have a child that has an error send a signal to the parent to break it out of the wait.)