- Problem: Parent script randomly spawns excessive number of worker scripts.
- Suspect: bash bug but unable to identify it
- OS: Ubuntu 16.04.03 LTS
- GNU bash: version 4.3.48(1)-release (x86_64-pc-linux-gnu)
- MySQL: 5.7.21 (MySQL repository)
What the parent script does is get data from MySQL and executes worker scripts in the background with the data from MySQL. The parent script is responsible for keeping no more than seven worker scripts running until all the data is processed. This was working flawlessly for years until a month or so ago. I suspect my problem is caused by a recent update. Here is the logic:
- Parent script gets data from MySQL server
- Parent script loops and launches worker scripts in the background passing data obtained from MySQL.
- Worker script spawns and writes a "lock file"
- Parent script keeps up to seven worker scripts running by monitoring the number of spawned worker scripts AND the number of worker lock files.
I know that the parent script can loop and spawn a number of child processes before the child process(s) has time to setup and write out it's lock file. That is why I maintain a child spawn count (SPWNCNT) to help avoid this scenario. Like I said, it use to work fine and now it doesn't.
Here is the portion of the script:
#!/bin/bash
......
###########################
# Loop through all unique codes and process them.
echo "$0: NOTICE: Started processing codes; `date`."
COMCNT=0
SPWNCNT=0
TPCNT=0
while read COMNUM
do
# Only permit a certain number of child processes
# so to not overload the machine or chew up to
# many MySQL connections.
PCNT=`ls -1 $TEMPDIR/*.Worker.lock 2>/dev/null | wc -l`
(( TPCNT = PCNT + SPWNCNT ))
TPCNT=`echo ${TPCNT#-}`
while [[ $TPCNT -gt 6 ]]
do
# Too many child processes.
sleep 1
PCNT=`ls -1 $TEMPDIR/*.Worker.lock 2>/dev/null | wc -l`
(( TPCNT = PCNT + SPWNCNT ))
TPCNT=`echo ${TPCNT#-}`
if [[ $SPWNCNT -gt 0 ]]
then
(( SPWNCNT = SPWNCNT - PCNT ))
if [[ $SPWNCNT -lt 0 ]]
then
SPWNCNT=0
fi
fi
done # while -gt 6
# Spawn a worker process
./Worker.sh $COMNUM &
(( SPWNCNT = SPWNCNT + 1 ))
(( COMCNT = COMCNT + 1 ))
if [ "$VERBOSE" = "true" ]
then
echo "$0: NOTICE: Spawned $COMNUM, count $COMCNT ($SPWNCNT); `date`"
fi
done << COMNUM_EOF
`cat $GEN_RATES_COMNUM_FILE | $MyCMD -B -N $MyDB`
COMNUM_EOF
......
And here is a debug output of the problem (using #!/bin/bash -x)
...... many lines showing same logic working correctly ......
++ wc -l
++ ls -1 '/tmp/*.Worker.lock'
+ PCNT=0
+ (( TPCNT = 0 + 7 ))
++ echo 7
+ TPCNT=7
+ [[ 7 -gt 0 ]]
+ (( SPWNCNT = 7 - 0 ))
+ [[ 7 -lt 0 ]]
+ [[ 7 -gt 6 ]]
+ sleep 1
++ wc -l
++ ls -1 /tmp/032500.Worker.lock /tmp/032800.Worker.lock
/tmp/033300.Worker.lock /tmp/033900.Worker.lock /tmp/034700.Worker.lock
/tmp/035400.Worker.lock /tmp/035600.Worker.lock /tmp/036000.Worker.lock
/tmp/036200.Worker.lock /tmp/036400.Worker.lock /tmp/036600.Worker.lock
/tmp/037000.Worker.lock /tmp/039100.Worker.lock /tmp/039600.Worker.lock
/tmp/040200.Worker.lock /tmp/040400.Worker.lock /tmp/041000.Worker.lock
/tmp/041200.Worker.lock /tmp/041600.Worker.lock /tmp/041800.Worker.lock
/tmp/042000.Worker.lock /tmp/043600.Worker.lock /tmp/046200.Worker.lock
/tmp/048600.Worker.lock /tmp/049600.Worker.lock /tmp/052000.Worker.lock
/tmp/052300.Worker.lock /tmp/054300.Worker.lock /tmp/054500.Worker.lock
/tmp/054900.Worker.lock /tmp/055300.Worker.lock /tmp/056000.Worker.lock
/tmp/056200.Worker.lock
/tmp/056600.Worker.lock /tmp/056900.Worker.lock /tmp/057800.Worker.lock
/tmp/058600.Worker.lock /tmp/060400.Worker.lock /tmp/060700.Worker.lock
+ PCNT=39
+ (( TPCNT = 39 + 7 ))
++ echo 46
+ TPCNT=46
+ [[ 7 -gt 0 ]]
+ (( SPWNCNT = 7 - 39 ))
+ [[ -32 -lt 0 ]]
+ SPWNCNT=0
+ [[ 46 -gt 6 ]]
+ sleep 1
++ wc -l
++ ls -1 /tmp/032500.Worker.lock /tmp/032800.Worker.lock
/tmp/033300.Worker.lock /tmp/033900.Worker.lock /tmp/034700.Worker.lock
/tmp/035400.Worker.lock /tmp/035600.Worker.lock /tmp/036000.Worker.lock
/tmp/036200.Worker.lock /tmp/036400.Worker.lock /tmp/036600.Worker.lock
/tmp/037000.Worker.lock /tmp/039100.Worker.lock /tmp/039600.Worker.lock
/tmp/040200.Worker.lock /tmp/040400.Worker.lock /tmp/041000.Worker.lock
/tmp/041200.Worker.lock /tmp/041600.Worker.lock /tmp/041800.Worker.lock
/tmp/042000.Worker.lock /tmp/043600.Worker.lock /tmp/046200.Worker.lock
/tmp/048600.Worker.lock /tmp/049600.Worker.lock /tmp/052000.Worker.lock
/tmp/052300.Worker.lock /tmp/054300.Worker.lock /tmp/054500.Worker.lock
/tmp/054900.Worker.lock /tmp/055300.Worker.lock /tmp/056000.Worker.lock
/tmp/056200.Worker.lock
/tmp/056600.Worker.lock /tmp/056900.Worker.lock /tmp/057800.Worker.lock
/tmp/058600.Worker.lock /tmp/060400.Worker.lock /tmp/060700.Worker.lock
+ PCNT=39
+ (( TPCNT = 39 + 0 ))
++ echo 39
+ TPCNT=39
+ [[ 0 -gt 0 ]]
+ [[ 39 -gt 6 ]]
+ sleep 1
So, how did the script jump from 7 running processes (TPCNT) (the proper target) to 39 and then to 46? I cannot see how the logic would permit this and the debug output doesn't appear to shed any light on this other than the bash shell just flaked out.
PCNT
) to the number of processes you've spawned (SPWNCNT
), but what quantity does that represent?ls
in your trace output. This seems to happen outside the script, the question is why?/tmp
contained no*Worker.lock
files before you started the script?*Worker.lock
files is flawed. It would be better for the parent script to keep track of children by adding$!
(pid of last background process) to an array. The array size tells you how many you've created. Then, either set up a FIFO for communication or test each pid withkill -0
. Create a new array with currently running pids as children finish.