6

I'm working with a program that outputs error messages when something goes wrong, but does not set its exit status accordingly: the exit status is always 0, indicating success. I'd like to run this program from a shell script and get a non-zero exit status if it emits any error messages, so the script can tell that the program failed.

The error messages follow a predictable pattern that I can match with grep, and grep sets its exit status based on whether it found a match, so I can pipe the program's output into grep and negate the result with ! to get the exit status I want.

The problem is that if I pipe into grep, I can't see the program's output anymore, because grep consumes it. I'd like to somehow scan the output for error messages but also display the output normally, since there are other important messages besides the errors. Unfortunately, grep doesn't have an option to pass-through all of its input, matching and non-matching lines alike.

One approach I've found that mostly works is:

! my_program | tee /dev/stderr | grep -q "error message"

This feeds the program's output into grep but also copies it to standard error, which isn't redirected, so I can see it. That's OK when my script's standard output and standard error both go to a terminal (so it doesn't matter which one gets the messages), but it can cause problems if standard out and standard error are redirected to different places; the messages will end up in the wrong place.

Using bash or POSIX shell, and GNU tools, is there a (concise) way to scan a program's standard out and/or standard error streams with something like grep and set exit status accordingly, but also let both streams go to their normal destinations?

(Note that my_program writes its error messages to standard out, not standard error, so a solution that can only scan the standard output stream is OK, though not ideal. And in case it makes a difference, I'm doing this on CentOS 7.)

4

...is there a (concise) way to scan a program's standard out and/or standard error streams with something like grep and set exit status accordingly, but also let both streams go to their normal destinations?

...my_program writes its error messages to standard out, not standard error, so a solution that can only scan the standard output stream is OK, though not ideal.

My solution answers the bolded portion above.

I think the easiest way to do this is through awk:

myprogram |
awk 'BEGIN {status = 0} /error message/ {status = 1} 1; END {exit(status)}'

The awk command outputs everything it inputs exactly as-is, but at the end it exits with a status dependent on whether "error message" was a part of the input.

Concise version (with a shorter variable name):

myprogram | awk 'BEGIN{s=0} /error message/{s=1} 1; END{exit(s)}'
  • Also note, I am fairly certain it is fundamentally impossible to simultaneously scan stdout and stderr, and send them subsequently to two separate places. Not to mention the fact that stderr is unbuffered and stdout is often buffered. – Wildcard Sep 3 '16 at 10:45
  • Correction to earlier comment: It is possible, but not with standard utilities. So, not fundamentally impossible. (To do so, you could use mkfifo to set up the stdout and stderr targets, then open them for reading using a custom-written program which will read from both sources and write each to the appropriate file descriptor, like a dual-action cat, and check for a pattern in each along the way.) – Wildcard Feb 1 '17 at 11:30
3

Here are some one liners that grep the error message $MSG on stdout/stderr, and do/do not stop the program as soon as the error message appears:

# grep on stdout, do not stop early
my_program | awk -v s="$MSG" '$0~s{r=1} 1; END{exit(r)}'

# grep on stdout, do stop early
my_program | awk -v s="$MSG" '$0~s{exit(1)} 1'

# grep on stderr, do not stop early
{ my_program 2>&1 >&3 | awk -v s="$MSG" '$0~s{r=1} 1; END{exit(r)}' >&2; } 3>&1

# grep on stderr, do stop early
{ my_program 2>&1 >&3 | awk -v s="$MSG" '$0~s{exit(1)} 1' >&2; } 3>&1

Notes:

  • For all: the stream on which you apply grep will lose its potential tty status, because my_program sees it going into awk. This might affect how my_program writes to this stream, e.g., it might not print rotating progress bars because it might assume it cannot control cursor position.

  • For all: stdout and stderr will not be merged, and can be redirected independently from each other, as usual.

  • For all: the original exit code of my_program gets completely ignored. The only other easy alternative is: exit with error if either my_program exits with error, or the error message appears. To get this behaviour, you need to enable pipefail in the Bash shell that executes the pipeline. E.g.:

    (set -o pipefail; my_program | awk -v s="$MSG" '$0~s{exit(1)} 1')
    
  • For grepping on stderr: the simple command lines above assume file descriptor 3 is unused. That's usually an ok assumption to make. To avoid this assumption, you can get Bash to allocate a file descriptor that's guaranteed to be unused with:

    bash -c 'exec {fd}>&1; { my_program 2>&1 >&${fd} | awk -v s="$MSG" '\''$0~s{exit(1)} 1'\'' >&2; } ${fd}>&1'
    
0

Try two terminals side by side. In the first shell window:

tail -F /tmp/xyzzy

In the second, just what you were doing, only to the tmp file:

my_program | tee /tmp/xyzzy | grep -q "error message"

Start them in that order.

It's klunky, as you'll want to clear the temp file every so often, or pick a new name, but it works.

Added later... Try something like:

my_program | tee /tmp/xyzzy ; grep -q "error message" /tmp/xyzzy

The exit status of the sequence is the exit status of the grep. Which is inverted from what you want, sigh. So negate it.

my_program | tee /tmp/xyzzy ; ! grep -q "error message" /tmp/xyzzy
  • Why use 2 terminals? Maybe I misread the question but I didn't see anything related mentioned. – Julie Pelletier Sep 3 '16 at 0:54
  • To separate the two kinds of output: important stuff being matched by grep, vs. everything. – John Aspinall Sep 3 '16 at 1:30
  • I need the output to actually come from the script that's running the program, like it would if the script just ran the program normally. (It's actually a Jenkins build script; Jenkins captures the script's output and saves it in a log. The exit status is so that Jenkins can tell whether the build has failed.) – Wyzard Sep 3 '16 at 1:30
  • Ahh, I see. Then something like my next answer. – John Aspinall Sep 3 '16 at 1:40
0

Make a demo function baz() that outputs "foo" to stdout and "bar" to stderr:

baz() { echo foo ; echo bar >& 2 ; }

Simple case, run grep foo on stdout, and grep bar on stderr:

{ baz 2>&1 1>&3 | grep bar 1>&2 ; } 3>&1 | grep foo

Same thing, but quietly, using tee; the output looks like no greps were used at all:

{ baz 2>&1 1>&3 | tee /dev/stderr | grep -q bar ; } 3>&1 | \
{ tee /dev/stderr | grep -q foo ; } 2>&1

Now add a conditional after grep -q bar, which prints "BEEP!" to stderr if bar is found:

{ baz 2>&1 1>&3 | tee /dev/stderr | grep -q bar && echo "BEEP!" >&2 ; } 3>&1 | \
{ tee /dev/stderr | grep -q foo ; } 2>&1

Output of the last two lines:

foo
bar
BEEP!
  • The redirect juggling needs more explanations, to be added later, time permitting... – agc Sep 3 '16 at 1:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.