One technique to control an errant pipeline is to count the number of pipeline processes after a while. If they haven't finished, then zap them and restart.
In this example code we use ask Bash for its process id (the
$$ variable), then use pgrep to find all sub-processes of the script. That is,
pgrep tells us all the workflow process IDs. If there are any, then we use the corresponding pkill command to squash them before a restart.
In addition, we use the date command to output a pretty log file, showing what the system is doing at what time.
# pipeline.sh -- start workflow; restart if jammed
while true; do
date '+%X workflow starting'
(sleep 30; echo gin) &
(sleep 30; echo tonic) &
date '+%X waiting for workflow to complete'
# count number of child procs (ie: workflow procs)
# If there are any, kill them then restart.
if pgrep -cP $$ > /dev/null ; then
date '+%X workflow jammed -- restarting; trying again'
pkill -P $$
date '+%X workflow done!'
start workflow control script
$ ./pipeline.sh &
02:06:39 PM workflow starting
02:06:39 PM waiting for workflow to complete
The pipeline scripts waits for a few seconds, but the workflow procs are still running, as they start with a "sleep 30"
Pipeline detects workflow is jammed, complains and kills them before a restart.
02:06:44 PM workflow jammed -- restarting; trying again
./pipeline.sh: line 27: 21293 Terminated ( sleep 30; echo gin )
./pipeline.sh: line 27: 21294 Terminated ( sleep 30; echo tonic )
02:06:46 PM workflow starting
02:06:46 PM waiting for workflow to complete
Pipeline is now waiting for workflow to complete. Here we'll cheat and manually make the pipeline complete by killing the workers.
$ pkill -f sleep
./pipeline.sh: line 27: 21363 Terminated sleep 30
./pipeline.sh: line 27: 21365 Terminated sleep 5
./pipeline.sh: line 27: 21364 Terminated sleep 30
Pipeline script now notices all workers are done, thus the pipeline is done. It make a concluding log message, then exits.
02:07:16 PM workflow done!
+ Done ./pipeline.sh