7

I have a bit of an issue with the following command which extracts a tar file & prints how many files have been extracted every second:

tar -xvf some_tar.tar -C a/directory | awk 'systime() > lasttime { lasttime = systime(); printf "%d files\n", NR; fflush(stdout) }'

Even if the tar command fails, the awk command will still return 0, which is undesirable because it does not reflect that the tar command failed.

How might I go about fixing this?

0

6 Answers 6

16

If you just want to see if any command in the pipeline failed, set the pipefail option. It's supported in ksh, zsh and Busybox (at least), in addition to Bash. With that option set, the exit status of a pipeline is the leftmost non-zero exit status returned by the commands involved.

$ set -o pipefail
$ (exit 123) | true
$ echo $?
123

Or with the pipeline just in a conditional (this should say "it failed"):

set -o pipefail
if false | true; then
    echo it succeeded
else
    echo it failed
fi
8

One approach in recent versions of bash would be to check the values in the PIPESTATUS array variable after you've invoked the tar|awk command pipeline. Per the bash man page:

PIPESTATUS
   An array variable (see Arrays below) containing a list of exit status
   values from the processes  in  the  most-recently-executed foreground
   pipeline (which may contain only a single command).

So the exit code from tar would be in ${PIPESTATUS[0]}, and the exit code from awk would be in ${PIPESTATUS[1]}.

2
  • Or turn on shopt/set -o pipefail and then $? will be the status of the tar (given the awk had status zero) Feb 23, 2023 at 17:05
  • To output whole array use echo ${PIPESTATUS[*]}
    – ks1322
    Feb 23, 2023 at 18:28
6

Don't use a pipeline at all. Use a named pipe.

mkfifo p
awk '...' < p &
tar -xvf some_tar.tar -C a/directory > p
echo $?

The awk command runs in the background, blocking until tar starts writing to the named pipe. Once tar exits and closes its end of the pipe, awk will exit after reading what is left to read from its end. The echo command will report tar's exit status, not awk's.

4
  • 1
    Elegant solution! Thank you. Feb 23, 2023 at 18:53
  • This has to be the best answer, it really uses *Nix's power!
    – Scottie H
    Feb 23, 2023 at 21:34
  • 1
    Actually, there's a bit of a problem I overlooked earlier. The point of running awk was to see its output; you can't do that when awk is running in the background. (Unless you run tar in the background as well, then put awk back in the foreground. But then you'll need to use wait to get the exit status of tar once awk exits, at which point this has ballooned way past the point where it's simpler than just using PIPESTATUS.)
    – chepner
    Feb 23, 2023 at 21:41
  • Doesn't seem to be a problem, output from background processes shows up fine for me Feb 24, 2023 at 1:33
5

You don't really need to. You can use the PIPESTATUS array to get the exit code from any of the programs in a pipeline. You can also add up all the exit codes in the array to derive a status for the entire pipeline (if the sum is non-zero then something in the pipeline failed).

From man bash:

PIPESTATUS

An array variable (see Arrays below) containing a list of exit status values from the processes in the most-recently-executed foreground pipeline (which may contain only a single command).

1
  • Summing the array elements in the shell, hmm... sum=$(tmp=("${PIPESTATUS[@]}"); IFS=+; eval echo "\$(( ${tmp[*]} ))")...
    – ilkkachu
    Feb 24, 2023 at 14:02
3

I expect calling systime() for every input line will slow down your pipeline significantly so the output from your pipeline won't accurately reflect how many files tar extracts per second. Consider only calling systime() once per thousand or million or something input lines or not calling systime() at all and just printing every thousand or million input lines if you want to see a progress indicator since you're not truly printing how may files have been extracted every second anyway.

Consider doing something like this to solve the problem you asked about and almost entirely remove the overhead of calling systime() (uses GNU awk for time functions and to have $0 in END contain the last line read and to be able to have NUL in the input):

{ tar -xvf some_tar.tar -C a/directory && printf '\0\n'; } |
awk -v n=1000000 '
    BEGIN { beg = systime() }
    NR%n == 0 { printf "%d files processed\n", NR }
    END {
        end = systime()
        if ( $0 == "\0" ) {
            numFiles = NR - 1
            exitStatus = 0
        }
        else {
            numFiles = NR
            exitStatus = 1
        }
        printf "%d files per sec\n", numFiles / (end > beg ? end - beg : 1)
        exit exitStatus
    }
'
4
  • Have you measured this? It seems unlikely that systime would use any significant time compared to extracting and writing a file to the filesystem. On modern systems it would not even need a syscall, but even if it did, one clock_gettime syscall costs maybe at most 5% of all the open, write, fstat, chmod, close, etc. syscalls extracting a file would entail (and likely more like 1%). Feb 23, 2023 at 13:31
  • 1
    @R..GitHubSTOPHELPINGICE - I hadn't when you asked but I just did a quick test and time { seq 10000000 | awk '{print $0}' >/dev/null; } output real 0m1.916s user 0m1.515s sys 0m0.000s while time { seq 10000000 | awk '{print systime(), $0}' >/dev/null; } output real 0m6.212s user 0m5.671s sys 0m0.015s. Feel free to try other things and post different timings if you like.
    – Ed Morton
    Feb 23, 2023 at 13:42
  • 2
    For me, the difference was about 2x with the systime and difference in volume of data written to /dev/null accounted fully for it. This was with GNU awk. Incidentally I also just discovered that busybox awk is flushing stdout after every print command, making it extremely slow for no apparent reason... Feb 23, 2023 at 13:56
  • 1
    @R..GitHubSTOPHELPINGICE good point on the extra data being written. When I changed it to $ time { seq 10000000 | awk '{systime(); print $0}' >/dev/null; } to remove the difference in volume of data written I got real 0m2.734s user 0m1.999s sys 0m0.000s so it took abut 1.5x the duration of the version without the systime() calls. I'm also using GNU awk,, version 5.2.1
    – Ed Morton
    Feb 23, 2023 at 21:20
2

I would suggest you add "2>&1" on the tar command, then, in the awk, move the print into the "END{...}" clause, with a conditional based on trapping the error message string.

OR, you could do something like

{ tar -xvf some_tar.tar -C a/directory ; echo "RC=$?" } | ...

and scan for that "RC=" condition and evaluate the result before doing the "printf".

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .