5

I am running a command and manipulating the output with my own script, but I don't want the main command to stop when my script fails.

For example:

a-command | tee logfile.txt | myscript

when my script fails midway, I look in the logfile.txt, I find it breaks where my script stopped working, but what I want is for the a-commmand to contninue and the logfile.txt to have the full log so that I can debug my script and fix the error.

What is the way to modify my command to treat the last pipeline part as optional or ignored in case of errors so that the first two parts (the command and the tee) to finish their jobs.

[EDIT] a-command is a lengthy task and my script is basically manipulating the output to better report status while the command is running. So I don't want to run my script after a-command finishes its work.

  • 2
    Just run them separately? a-command > logfile.txt & myscript < logfile.txt – muru Jun 18 at 7:03
  • a-command is a lengthy task and my script is basically manipulating the output to better report status while the command is running. – Bishoy Jun 19 at 21:45
  • Ok. How does that affect things? – muru Jun 20 at 0:11
10
a-command | tee logfile.txt | { myscript; cat >/dev/null; }

This would run your pipeline as usual at first, until myscript terminates (for whatever reason). At that point, cat would take over reading from tee until there in no more data arriving. The data read by cat is discarded by piping it to /dev/null.

If a-command finishes without myscript ending/failing first, myscript would fail to read more data and would presumably terminate (?). At the point when myscript terminates, cat is started, but as there is no more data to read, it would immediately terminate and the pipeline would be done.


Addressing TooTea's comment about making sure that we still get the correct exit status for the pipeline:

a-command | tee logfile.txt | ( myscript; err=$?; cat >/dev/null; exit "$err" )
| improve this answer | |
  • This has the side effect that the exit status of the whole pipeline is no longer the exit status of myscript but that of cat instead (likely always 0). Of course it's easily fixable with a little $?-juggling inside the curly braces. – TooTea Jun 18 at 20:12
  • @TooTea I've now addressed that too. – Kusalananda Jun 18 at 21:28
  • When I tried the command the ends with exit "$err", I found that my ssh session gets closed at the end of running the command. :D – Bishoy Jun 22 at 22:32
  • @Bishoy My bad, fixed now. – Kusalananda Jun 23 at 0:11
10

Tee (in Linux) has an option that ignores pipe failures.

a-command | tee --output-error=warn logfile.txt | myscript

When myscript fails or is killed, a-command continues to run and the log continues to grow.

You can rerun your script, and have it exit when it catches up the last complete block of the log:

myscript < logfile.txt

You can rerun your script, and have it wait for additions when it catches up.

tail -999999f < logfile.txt | myscript

A more complex example, contained in a Bash script.

logger represents your a-command. It generates 36 permutations of a short string, one per second. All the output is teed to 593580.log.

awk represents your "myscript". It prints a subset of the input.

wdog is my watchdog utility. -d 5 makes it debug its actions. -t 25 makes it timeout the process under control (the awk) after 25 seconds, with a SIGUSR1. This just saves me manually running a kill to simulate your script failure -- I like repeatable tests.

When the awk goes away, the cat in the same compound command gets to read the pipe, and copies the remaining data to a duplicate log. So you can re-run your script against the full log, or the unprocessed data only, and you can compare the two logs to find exactly where you crashed.

Alternatively, you can cat >/dev/null, just to keep the pipe alive so logger continues to run.

Both the logfile copies seem to be line-buffered: tail -f shows then in real-time.

The example script:

#! /bin/bash

logger () {

    for Q in {0..1}{A..C}{A..F}; do
        printf '%s\n' "${Q}"
        sleep 1
    done
}
    
AWK='
/C/ { printf ("awk %d %s\n", NR, $0); }
'

    logger | tee 593580.log | 
        { 
            date
            wdog -d 5 -t 25 awk "${AWK}"
            date
            cat > 593580.add 
            date
        }

The test run:

paul $ ./593580
Thu 18 Jun 15:35:24 BST 2020
wdog       25.000| Thu Jun 18 15:35:49.574 2020
wdog 15:35:24.574| Started awk as 14035
awk 3 0AC
wdog 15:35:29.579| Tick
awk 9 0BC
wdog 15:35:34.583| Tick
awk 13 0CA
awk 14 0CB
awk 15 0CC
wdog 15:35:39.586| Tick
awk 16 0CD
awk 17 0CE
awk 18 0CF
wdog 15:35:44.591| Tick
awk 21 1AC
wdog 15:35:49.579| Tick
wdog 15:35:49.579| Timed out child 14035 with signal 10
wdog 15:35:49.580| Child 14035 terminated with signal 10
Thu 18 Jun 15:35:49 BST 2020
Thu 18 Jun 15:36:00 BST 2020
paul $ 
| improve this answer | |
  • Thanks, --output-error was exactly what I was looking for. The rest of tips and the example are very useful though. I am curious though if there is another workaround instead of --output-error, maybe involving && or || or similar bash features? Just in case the command in the middle is not tee. – Bishoy Jun 19 at 21:41
  • I believe both Kusalananda's answer and mine will work for any pipeline, not just after tee. All that matters is that the failing process is in a compound statement { here } where there is an alternative reader for stdin. In mine, there are five processes, all of which inherit the pipe as stdin from the { } construct. The three date commands don't read stdin, the awk gets first contact, and anything not read by awk is available to the next command. As long as something holds stdin, the pipe will stay active. Of course, if you put in sleep 1000, the pipe fills and a-cmd suspends. – Paul_Pedant Jun 20 at 10:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.