How to cause awk to fail if previous command (tar) also fails?

Question

I have a bit of an issue with the following command which extracts a tar file & prints how many files have been extracted every second:

tar -xvf some_tar.tar -C a/directory | awk 'systime() > lasttime { lasttime = systime(); printf "%d files\n", NR; fflush(stdout) }'

Even if the tar command fails, the awk command will still return 0, which is undesirable because it does not reflect that the tar command failed.

How might I go about fixing this?

Answer 1

If you just want to see if any command in the pipeline failed, set the pipefail option. It's supported in ksh, zsh and Busybox (at least), in addition to Bash. With that option set, the exit status of a pipeline is the leftmost non-zero exit status returned by the commands involved.

$ set -o pipefail
$ (exit 123) | true
$ echo $?
123

Or with the pipeline just in a conditional (this should say "it failed"):

set -o pipefail
if false | true; then
    echo it succeeded
else
    echo it failed
fi

Answer 2

One approach in recent versions of bash would be to check the values in the PIPESTATUS array variable after you've invoked the tar|awk command pipeline. Per the bash man page:

PIPESTATUS
   An array variable (see Arrays below) containing a list of exit status
   values from the processes  in  the  most-recently-executed foreground
   pipeline (which may contain only a single command).

So the exit code from tar would be in ${PIPESTATUS[0]}, and the exit code from awk would be in ${PIPESTATUS[1]}.

Answer 3

Don't use a pipeline at all. Use a named pipe.

mkfifo p
awk '...' < p &
tar -xvf some_tar.tar -C a/directory > p
echo $?

The awk command runs in the background, blocking until tar starts writing to the named pipe. Once tar exits and closes its end of the pipe, awk will exit after reading what is left to read from its end. The echo command will report tar's exit status, not awk's.

Answer 4

You don't really need to. You can use the PIPESTATUS array to get the exit code from any of the programs in a pipeline. You can also add up all the exit codes in the array to derive a status for the entire pipeline (if the sum is non-zero then something in the pipeline failed).

From man bash:

PIPESTATUS

An array variable (see Arrays below) containing a list of exit status values from the processes in the most-recently-executed foreground pipeline (which may contain only a single command).

Answer 5

I expect calling systime() for every input line will slow down your pipeline significantly so the output from your pipeline won't accurately reflect how many files tar extracts per second. Consider only calling systime() once per thousand or million or something input lines or not calling systime() at all and just printing every thousand or million input lines if you want to see a progress indicator since you're not truly printing how may files have been extracted every second anyway.

Consider doing something like this to solve the problem you asked about and almost entirely remove the overhead of calling systime() (uses GNU awk for time functions and to have $0 in END contain the last line read and to be able to have NUL in the input):

{ tar -xvf some_tar.tar -C a/directory && printf '\0\n'; } |
awk -v n=1000000 '
    BEGIN { beg = systime() }
    NR%n == 0 { printf "%d files processed\n", NR }
    END {
        end = systime()
        if ( $0 == "\0" ) {
            numFiles = NR - 1
            exitStatus = 0
        }
        else {
            numFiles = NR
            exitStatus = 1
        }
        printf "%d files per sec\n", numFiles / (end > beg ? end - beg : 1)
        exit exitStatus
    }
'

Answer 6

I would suggest you add "2>&1" on the tar command, then, in the awk, move the print into the "END{...}" clause, with a conditional based on trapping the error message string.

OR, you could do something like

{ tar -xvf some_tar.tar -C a/directory ; echo "RC=$?" } | ...

and scan for that "RC=" condition and evaluate the result before doing the "printf".