2

I have the following, I thought it ran the subshells in parallel, but it actually seems to run them in series, and I cannot figure out why:

#!/usr/bin/env bash

set -e;
set -m # allow for job control


EXIT_CODE=0;  # exit code of overall script

function handleJobs() {
     for job in `jobs -p`; do
         echo "PID => ${job}"
         CODE=0;
         wait ${job} || CODE=$?
         if [[ "${CODE}" != "0" ]]; then
            echo "At least one process failed with exit code => ${CODE}" ;
            EXIT_CODE=1;
         fi
     done
}

trap 'handleJobs' CHLD


for file in "$HOME/mongodump_dev/cdt_db/"* ; do

  file="$(basename "$file")"

  if [[ "$file" != "system"* && "$file" != "locks"* ]]; then

    mongorestore \
        --db "cdt_dev" \
        --collection "${file%.*}" \
        --host "<my-host>" \
        "$HOME/mongodump_dev/cdt_db/$file"  &

  fi;

done


wait;
echo "exit code => $EXIT_CODE"
exit "$EXIT_CODE"

Anyone know why the subshells are running in series?

I tried doing this instead:

(
        mongorestore \
            --db "cdt_dev" \
            --collection "${file%.*}" \
            --host "<my-host>" \
            "$HOME/mongodump_dev/cdt_db/$file"  &

) &

now they run in parallel, but now the whole script will never exit, and I fear that I am actually not accurately capturing the exit code.

  • What makes you think they're running in series? – Barmar Feb 21 '18 at 21:58
  • without the modification using (...) &, it's clear that it's running in series from the logs and the time it takes, when in parallel, the logs are very different and it's much faster – Alexander Mills Feb 21 '18 at 22:01
  • have you tried using ps to see how many mongorestore processes are actually running? – Barmar Feb 21 '18 at 22:04
  • i think the logs are best, not sure how ps is going to tell me if any processes are overlapping in time – Alexander Mills Feb 21 '18 at 22:15
  • If they're executing in parallel, ps should show all of them. – Barmar Feb 21 '18 at 22:18
2

You have a number of "issues" with your script as written. What I did was to copy your script, and for the purposes of testing and evaluation, simplify it by doing thing s like replacing mongorestore with sleep 9999, and by inserting echo trace message that append to a log file. The long and short of it is that it doesn't seem to me that you need any traps; just replace your mongorestore statement with:

( mongorestore ...
  exit_code=$?
  [ $exit_code -ne 0 ] \
  && printf "Process %s for file %s failed with exit code %s\n" \
       $BASHPID  "${file%.*}" $exit_code
) &

BTW, a simple way to verify that all your sub-processes have been spawned in parallel is open another terminal and run pgrep -a mongorestore.

Also, your use of EXIT_CODE at the end of your script seems to me should never have a value in it, so your exit command should fail with some kind of error, which would mean your script would always exit with an error.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.