0
  • Problem: Parent script randomly spawns excessive number of worker scripts.
  • Suspect: bash bug but unable to identify it
  • OS: Ubuntu 16.04.03 LTS
  • GNU bash: version 4.3.48(1)-release (x86_64-pc-linux-gnu)
  • MySQL: 5.7.21 (MySQL repository)

What the parent script does is get data from MySQL and executes worker scripts in the background with the data from MySQL. The parent script is responsible for keeping no more than seven worker scripts running until all the data is processed. This was working flawlessly for years until a month or so ago. I suspect my problem is caused by a recent update. Here is the logic:

  1. Parent script gets data from MySQL server
  2. Parent script loops and launches worker scripts in the background passing data obtained from MySQL.
  3. Worker script spawns and writes a "lock file"
  4. Parent script keeps up to seven worker scripts running by monitoring the number of spawned worker scripts AND the number of worker lock files.

I know that the parent script can loop and spawn a number of child processes before the child process(s) has time to setup and write out it's lock file. That is why I maintain a child spawn count (SPWNCNT) to help avoid this scenario. Like I said, it use to work fine and now it doesn't.

Here is the portion of the script:

#!/bin/bash

......


###########################
# Loop through all unique codes and process them.
echo "$0: NOTICE: Started processing codes; `date`."
COMCNT=0
SPWNCNT=0
TPCNT=0
while read COMNUM
do
  # Only permit a certain number of child processes
  # so to not overload the machine or chew up to
  # many MySQL connections.
  PCNT=`ls -1 $TEMPDIR/*.Worker.lock 2>/dev/null | wc -l`
  (( TPCNT = PCNT + SPWNCNT ))
  TPCNT=`echo ${TPCNT#-}`
  while [[ $TPCNT -gt 6 ]]
  do
    # Too many child processes.
    sleep 1
    PCNT=`ls -1 $TEMPDIR/*.Worker.lock 2>/dev/null | wc -l`
    (( TPCNT = PCNT + SPWNCNT ))
    TPCNT=`echo ${TPCNT#-}`

    if [[ $SPWNCNT -gt 0 ]]
    then
      (( SPWNCNT = SPWNCNT - PCNT ))
      if [[ $SPWNCNT -lt 0 ]]
      then
        SPWNCNT=0
      fi
    fi

  done # while -gt 6

  # Spawn a worker process
  ./Worker.sh $COMNUM &
  (( SPWNCNT = SPWNCNT + 1 ))
  (( COMCNT = COMCNT + 1 ))

  if [ "$VERBOSE" = "true" ]
  then
    echo "$0: NOTICE: Spawned $COMNUM, count $COMCNT ($SPWNCNT); `date`"
  fi

done << COMNUM_EOF
  `cat $GEN_RATES_COMNUM_FILE | $MyCMD -B -N $MyDB`
COMNUM_EOF

......

And here is a debug output of the problem (using #!/bin/bash -x)

...... many lines showing same logic working correctly ......
++ wc -l
++ ls -1 '/tmp/*.Worker.lock'
+ PCNT=0
+ ((  TPCNT = 0 + 7  ))
++ echo 7
+ TPCNT=7
+ [[ 7 -gt 0 ]]
+ ((  SPWNCNT = 7 - 0  ))
+ [[ 7 -lt 0 ]]
+ [[ 7 -gt 6 ]]
+ sleep 1
++ wc -l
++ ls -1 /tmp/032500.Worker.lock /tmp/032800.Worker.lock 
/tmp/033300.Worker.lock /tmp/033900.Worker.lock /tmp/034700.Worker.lock 
/tmp/035400.Worker.lock /tmp/035600.Worker.lock /tmp/036000.Worker.lock 
/tmp/036200.Worker.lock /tmp/036400.Worker.lock /tmp/036600.Worker.lock 
/tmp/037000.Worker.lock /tmp/039100.Worker.lock /tmp/039600.Worker.lock 
/tmp/040200.Worker.lock /tmp/040400.Worker.lock /tmp/041000.Worker.lock 
/tmp/041200.Worker.lock /tmp/041600.Worker.lock /tmp/041800.Worker.lock 
/tmp/042000.Worker.lock /tmp/043600.Worker.lock /tmp/046200.Worker.lock 
/tmp/048600.Worker.lock /tmp/049600.Worker.lock /tmp/052000.Worker.lock 
/tmp/052300.Worker.lock /tmp/054300.Worker.lock /tmp/054500.Worker.lock 
/tmp/054900.Worker.lock /tmp/055300.Worker.lock /tmp/056000.Worker.lock 
/tmp/056200.Worker.lock
/tmp/056600.Worker.lock /tmp/056900.Worker.lock /tmp/057800.Worker.lock 
/tmp/058600.Worker.lock /tmp/060400.Worker.lock /tmp/060700.Worker.lock
+ PCNT=39
+ ((  TPCNT = 39 + 7  ))
++ echo 46
+ TPCNT=46
+ [[ 7 -gt 0 ]]
+ ((  SPWNCNT = 7 - 39  ))
+ [[ -32 -lt 0 ]]
+ SPWNCNT=0
+ [[ 46 -gt 6 ]]
+ sleep 1
++ wc -l
++ ls -1 /tmp/032500.Worker.lock /tmp/032800.Worker.lock 
/tmp/033300.Worker.lock /tmp/033900.Worker.lock /tmp/034700.Worker.lock 
/tmp/035400.Worker.lock /tmp/035600.Worker.lock /tmp/036000.Worker.lock 
/tmp/036200.Worker.lock /tmp/036400.Worker.lock /tmp/036600.Worker.lock 
/tmp/037000.Worker.lock /tmp/039100.Worker.lock /tmp/039600.Worker.lock 
/tmp/040200.Worker.lock /tmp/040400.Worker.lock /tmp/041000.Worker.lock 
/tmp/041200.Worker.lock /tmp/041600.Worker.lock /tmp/041800.Worker.lock 
/tmp/042000.Worker.lock /tmp/043600.Worker.lock /tmp/046200.Worker.lock 
/tmp/048600.Worker.lock /tmp/049600.Worker.lock /tmp/052000.Worker.lock 
/tmp/052300.Worker.lock /tmp/054300.Worker.lock /tmp/054500.Worker.lock 
/tmp/054900.Worker.lock /tmp/055300.Worker.lock /tmp/056000.Worker.lock 
/tmp/056200.Worker.lock
/tmp/056600.Worker.lock /tmp/056900.Worker.lock /tmp/057800.Worker.lock 
/tmp/058600.Worker.lock /tmp/060400.Worker.lock /tmp/060700.Worker.lock
+ PCNT=39
+ ((  TPCNT = 39 + 0  ))
++ echo 39
+ TPCNT=39
+ [[ 0 -gt 0 ]]
+ [[ 39 -gt 6 ]]
+ sleep 1

So, how did the script jump from 7 running processes (TPCNT) (the proper target) to 39 and then to 46? I cannot see how the logic would permit this and the debug output doesn't appear to shed any light on this other than the bash shell just flaked out.

  • We can't see where the lock files get created here, and I'm not exactly sure how the logic is supposed to work either: you add the number of lockfiles (PCNT) to the number of processes you've spawned (SPWNCNT), but what quantity does that represent? – ilkkachu Jan 26 '18 at 13:19
  • For whatever reason there are 39 lock files created between the first and the second ls in your trace output. This seems to happen outside the script, the question is why? – nohillside Jan 26 '18 at 14:06
  • The lock file is created in the worker script and is the first command it will execute: .. echo $$ > /tmp/$1.Worker.lock .. That is not the problem. The problem is why does the parent all the sudden spawn 30+ more workers? If TPCNT ever exceeds 6 then the script should cool down and wait. The only way the workers can spawn is if the parent spawns them and that is what is happening. I do not understand why - especially when it loops correctly 100+ times before it flakes out. – Van Jan 26 '18 at 15:01
  • Are you sure that /tmp contained no *Worker.lock files before you started the script? – Andy Dalton Jan 26 '18 at 15:47
  • 1
    I think the approach of spawning workers and having those create *Worker.lock files is flawed. It would be better for the parent script to keep track of children by adding $! (pid of last background process) to an array. The array size tells you how many you've created. Then, either set up a FIFO for communication or test each pid with kill -0. Create a new array with currently running pids as children finish. – m0dular Jan 26 '18 at 20:12
1

For those who find this post later in life.... Here is the solution that worked for me based on m0dular's response above. m0dular actually deserves the credit.

#!/bin/bash

.....

# Intialize the Child Process List (array)
CPLIST=()

# Max concurrent child processes
MAXCP=6

# Worker loop to spawn and monitor child (worker) processes
while [[ SOME-CONDITION ]]
do

  # Monitor Child Process List (array)
  # Ensure that we don't exceed Max Child Processes
  if [[ ${#CPLIST[@]} -gt $MAXCP ]]
  then
    while [[ ${#CPLIST[@]} -gt $MAXCP ]]
    do
      sleep 1
      # Check each child processes to see if it's still running.
      for idx in ${!CPLIST[@]}
      do
        # Is child process still alive?
        kill -0 ${CPLIST[$idx]} 2>/dev/null
        if [[ $? -gt 0 ]]
        then
          # Child process is no longer running.
          # Remove it from the child process list (array).
          unset CPLIST[$idx]
        fi # if $?
      done # for idx
    done # while MAXCP
  fi # if MAXCP

  # Spawn a child process
  ./MyProgram &
  # Append Child Process PID to Child Process List
  CPLIST=(${CPLIST[@]} $!)

done # while

.....

# (end of file)
  • 1
    Glad to help. I didn't submit a real answer, just a comment. – m0dular Jan 29 '18 at 17:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.