BASH loops, counters, child processes; counter not working

Question

• Problem: Parent script randomly spawns excessive number of worker scripts.
• Suspect: bash bug but unable to identify it
• OS: Ubuntu 16.04.03 LTS
• GNU bash: version 4.3.48(1)-release (x86_64-pc-linux-gnu)
• MySQL: 5.7.21 (MySQL repository)

What the parent script does is get data from MySQL and executes worker scripts in the background with the data from MySQL. The parent script is responsible for keeping no more than seven worker scripts running until all the data is processed. This was working flawlessly for years until a month or so ago. I suspect my problem is caused by a recent update. Here is the logic:

1. Parent script gets data from MySQL server
2. Parent script loops and launches worker scripts in the background passing data obtained from MySQL.
3. Worker script spawns and writes a "lock file"
4. Parent script keeps up to seven worker scripts running by monitoring the number of spawned worker scripts AND the number of worker lock files.

I know that the parent script can loop and spawn a number of child processes before the child process(s) has time to setup and write out it's lock file. That is why I maintain a child spawn count (SPWNCNT) to help avoid this scenario. Like I said, it use to work fine and now it doesn't.

Here is the portion of the script:

#!/bin/bash

......


###########################
# Loop through all unique codes and process them.
echo "$0: NOTICE: Started processing codes; `date`."
COMCNT=0
SPWNCNT=0
TPCNT=0
while read COMNUM
do
  # Only permit a certain number of child processes
  # so to not overload the machine or chew up to
  # many MySQL connections.
  PCNT=`ls -1 $TEMPDIR/*.Worker.lock 2>/dev/null | wc -l`
  (( TPCNT = PCNT + SPWNCNT ))
  TPCNT=`echo ${TPCNT#-}`
  while [[ $TPCNT -gt 6 ]]
  do
    # Too many child processes.
    sleep 1
    PCNT=`ls -1 $TEMPDIR/*.Worker.lock 2>/dev/null | wc -l`
    (( TPCNT = PCNT + SPWNCNT ))
    TPCNT=`echo ${TPCNT#-}`

    if [[ $SPWNCNT -gt 0 ]]
    then
      (( SPWNCNT = SPWNCNT - PCNT ))
      if [[ $SPWNCNT -lt 0 ]]
      then
        SPWNCNT=0
      fi
    fi

  done # while -gt 6

  # Spawn a worker process
  ./Worker.sh $COMNUM &
  (( SPWNCNT = SPWNCNT + 1 ))
  (( COMCNT = COMCNT + 1 ))

  if [ "$VERBOSE" = "true" ]
  then
    echo "$0: NOTICE: Spawned $COMNUM, count $COMCNT ($SPWNCNT); `date`"
  fi

done << COMNUM_EOF
  `cat $GEN_RATES_COMNUM_FILE | $MyCMD -B -N $MyDB`
COMNUM_EOF

......

And here is a debug output of the problem (using #!/bin/bash -x)

...... many lines showing same logic working correctly ......
++ wc -l
++ ls -1 '/tmp/*.Worker.lock'
+ PCNT=0
+ ((  TPCNT = 0 + 7  ))
++ echo 7
+ TPCNT=7
+ [[ 7 -gt 0 ]]
+ ((  SPWNCNT = 7 - 0  ))
+ [[ 7 -lt 0 ]]
+ [[ 7 -gt 6 ]]
+ sleep 1
++ wc -l
++ ls -1 /tmp/032500.Worker.lock /tmp/032800.Worker.lock 
/tmp/033300.Worker.lock /tmp/033900.Worker.lock /tmp/034700.Worker.lock 
/tmp/035400.Worker.lock /tmp/035600.Worker.lock /tmp/036000.Worker.lock 
/tmp/036200.Worker.lock /tmp/036400.Worker.lock /tmp/036600.Worker.lock 
/tmp/037000.Worker.lock /tmp/039100.Worker.lock /tmp/039600.Worker.lock 
/tmp/040200.Worker.lock /tmp/040400.Worker.lock /tmp/041000.Worker.lock 
/tmp/041200.Worker.lock /tmp/041600.Worker.lock /tmp/041800.Worker.lock 
/tmp/042000.Worker.lock /tmp/043600.Worker.lock /tmp/046200.Worker.lock 
/tmp/048600.Worker.lock /tmp/049600.Worker.lock /tmp/052000.Worker.lock 
/tmp/052300.Worker.lock /tmp/054300.Worker.lock /tmp/054500.Worker.lock 
/tmp/054900.Worker.lock /tmp/055300.Worker.lock /tmp/056000.Worker.lock 
/tmp/056200.Worker.lock
/tmp/056600.Worker.lock /tmp/056900.Worker.lock /tmp/057800.Worker.lock 
/tmp/058600.Worker.lock /tmp/060400.Worker.lock /tmp/060700.Worker.lock
+ PCNT=39
+ ((  TPCNT = 39 + 7  ))
++ echo 46
+ TPCNT=46
+ [[ 7 -gt 0 ]]
+ ((  SPWNCNT = 7 - 39  ))
+ [[ -32 -lt 0 ]]
+ SPWNCNT=0
+ [[ 46 -gt 6 ]]
+ sleep 1
++ wc -l
++ ls -1 /tmp/032500.Worker.lock /tmp/032800.Worker.lock 
/tmp/033300.Worker.lock /tmp/033900.Worker.lock /tmp/034700.Worker.lock 
/tmp/035400.Worker.lock /tmp/035600.Worker.lock /tmp/036000.Worker.lock 
/tmp/036200.Worker.lock /tmp/036400.Worker.lock /tmp/036600.Worker.lock 
/tmp/037000.Worker.lock /tmp/039100.Worker.lock /tmp/039600.Worker.lock 
/tmp/040200.Worker.lock /tmp/040400.Worker.lock /tmp/041000.Worker.lock 
/tmp/041200.Worker.lock /tmp/041600.Worker.lock /tmp/041800.Worker.lock 
/tmp/042000.Worker.lock /tmp/043600.Worker.lock /tmp/046200.Worker.lock 
/tmp/048600.Worker.lock /tmp/049600.Worker.lock /tmp/052000.Worker.lock 
/tmp/052300.Worker.lock /tmp/054300.Worker.lock /tmp/054500.Worker.lock 
/tmp/054900.Worker.lock /tmp/055300.Worker.lock /tmp/056000.Worker.lock 
/tmp/056200.Worker.lock
/tmp/056600.Worker.lock /tmp/056900.Worker.lock /tmp/057800.Worker.lock 
/tmp/058600.Worker.lock /tmp/060400.Worker.lock /tmp/060700.Worker.lock
+ PCNT=39
+ ((  TPCNT = 39 + 0  ))
++ echo 39
+ TPCNT=39
+ [[ 0 -gt 0 ]]
+ [[ 39 -gt 6 ]]
+ sleep 1

So, how did the script jump from 7 running processes (TPCNT) (the proper target) to 39 and then to 46? I cannot see how the logic would permit this and the debug output doesn't appear to shed any light on this other than the bash shell just flaked out.

Answer 1

For those who find this post later in life.... Here is the solution that worked for me based on m0dular's response above. m0dular actually deserves the credit.

#!/bin/bash

.....

# Intialize the Child Process List (array)
CPLIST=()

# Max concurrent child processes
MAXCP=6

# Worker loop to spawn and monitor child (worker) processes
while [[ SOME-CONDITION ]]
do

  # Monitor Child Process List (array)
  # Ensure that we don't exceed Max Child Processes
  if [[ ${#CPLIST[@]} -gt $MAXCP ]]
  then
    while [[ ${#CPLIST[@]} -gt $MAXCP ]]
    do
      sleep 1
      # Check each child processes to see if it's still running.
      for idx in ${!CPLIST[@]}
      do
        # Is child process still alive?
        kill -0 ${CPLIST[$idx]} 2>/dev/null
        if [[ $? -gt 0 ]]
        then
          # Child process is no longer running.
          # Remove it from the child process list (array).
          unset CPLIST[$idx]
        fi # if $?
      done # for idx
    done # while MAXCP
  fi # if MAXCP

  # Spawn a child process
  ./MyProgram &
  # Append Child Process PID to Child Process List
  CPLIST=(${CPLIST[@]} $!)

done # while

.....

# (end of file)