while true; do 
    # process substitution instead of usual pipeline to avoid waiting for all programs
    handle_input_with_timeout < <( prog1 | prog2 | prog3 )
    echo "Data stopped flowing. Trying again"
    # perminate stuck programs in previous pipeline, then try again
    sleep 5
done

How to reliably get rid of prog1, prog2 and prog3 that can get stuck and hold resources necessary to try again?

Can it be done in Bash alone or I need to use cgroups?

  • See Best way to kill all child processes – terdon May 9 '14 at 13:40
  • @terdon, Sending SIGKILL to the process group that runs pipeline also kills bash running this script. All these programs appear in one process group. How to easily move them to separate process group? – Vi. May 9 '14 at 14:25
  • Found one solution... – Vi. May 9 '14 at 14:48
  • I was thinking you could target the while loop's group. Not the parent shell. – terdon May 9 '14 at 14:48

You can use timeout include in GNU's coreutils:

timeout <time in second> bash -c "prog1 | prog2 | prog3"

Example:

timeout 5 bash -c "pwd | sleep 10"

Use time to make sure it works:

$ time timeout 5 bash -c "pwd | sleep 10"

real    0m5.003s
user    0m0.000s
sys     0m0.000s
  • The timeout should be extended each time something appears on stdin. It should work like watchdog: data is flowing => OK, data interrupted => restart the chain. – Vi. May 9 '14 at 14:08
  • @Vi.: How do you know data is OK or interrupted? – cuonglm May 9 '14 at 14:16
  • handle_input_with_timeout sets up alarm(2) and rewinds it each time new data appears on stdin. If SIGALRM comes, it prints "timeout" and exits. – Vi. May 9 '14 at 14:18
  • Why don't you use timeout instead of alarm() in your handle_input_with_timeout function? – cuonglm May 9 '14 at 14:58
  • What timeout? A program (timeout(1))? It's not for this use case. Or you mean starting timeout(1) to each line of input data? alarm(2) and SIGALRM is easy to set up in Perl... – Vi. May 9 '14 at 15:01
up vote 4 down vote accepted

This can be done with process groups (suggested here) and setsid to start new one:

while true; do 

    handle_input_with_timeout < <( setsid bash -c '
          printf -- "-$$" > /tmp/saved_process_group.pid
          prog1 | prog2 | prog3
          ')
    echo "Data stopped flowing. Trying again"
    kill -9 $(< /tmp/saved_process_group.pid )

    sleep 5
done

One technique to control an errant pipeline is to count the number of pipeline processes after a while. If they haven't finished, then zap them and restart.

In this example code we use ask Bash for its process id (the $$ variable), then use pgrep to find all sub-processes of the script. That is, pgrep tells us all the workflow process IDs. If there are any, then we use the corresponding pkill command to squash them before a restart.

In addition, we use the date command to output a pretty log file, showing what the system is doing at what time.

source: https://github.com/shavenwarthog/johntellsall/tree/master/karma/kill-pipeline


#!/bin/bash

# pipeline.sh -- start workflow; restart if jammed

while true; do

    date '+%X workflow starting'
    (sleep 30; echo gin) &
    (sleep 30; echo tonic) &

    date '+%X waiting for workflow to complete'
    sleep 5

    # count number of child procs (ie: workflow procs)
    # If there are any, kill them then restart.
    if pgrep -cP $$ > /dev/null ; then 
        date '+%X workflow jammed -- restarting; trying again'
        pkill -P $$
        sleep 2
        continue
    fi

    date '+%X workflow done!'
    break

done

Test run:

start workflow control script

$ ./pipeline.sh &

[1] 21291

02:06:39 PM workflow starting

02:06:39 PM waiting for workflow to complete

The pipeline scripts waits for a few seconds, but the workflow procs are still running, as they start with a "sleep 30"

Pipeline detects workflow is jammed, complains and kills them before a restart.

02:06:44 PM workflow jammed -- restarting; trying again

./pipeline.sh: line 27: 21293 Terminated ( sleep 30; echo gin )

./pipeline.sh: line 27: 21294 Terminated ( sleep 30; echo tonic )

02:06:46 PM workflow starting

02:06:46 PM waiting for workflow to complete

Pipeline is now waiting for workflow to complete. Here we'll cheat and manually make the pipeline complete by killing the workers.

$ pkill -f sleep

./pipeline.sh: line 27: 21363 Terminated sleep 30

./pipeline.sh: line 27: 21365 Terminated sleep 5

./pipeline.sh: line 27: 21364 Terminated sleep 30

tonic

Pipeline script now notices all workers are done, thus the pipeline is done. It make a concluding log message, then exits.

02:07:16 PM workflow done!

[1]+ Done ./pipeline.sh

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.