2

In my bash scripts, I often use pipes and would like to know which stage of the pipe was causing the problem in case of errors. The basic structure of such snippets is:

#!/bin/bash

ProduceCommand 2>/dev/null | ConsumeCommand >/dev/null 2>&1
PipeErrors=("${PIPESTATUS[@]}")
[[ "${PipeErrors[0]}" -eq '0' ]] || { HandleErrorInProduceCommand; }
[[ "${PipeErrors[1]}" -eq '0' ]] || { HandleErrorInConsumeCommand; }

Now (interestingly enough for the first time) I am in a situation where it would be great if I could use either tee or pee. But what happens to $PIPESTATUS when using these commands? For example:

#!/bin/bash

ProduceCommand 2>/dev/null | tee >(ConsumeCommand1) >(ConsumeCommand2) >/dev/null 2>&1
PipeErrors=("${PIPESTATUS[@]}")

or

#!/bin/bash

ProduceCommand 2>/dev/null | pee ConsumeCommand1 ConsumeCommand2 2>/dev/null
PipeErrors=("${PIPESTATUS[@]}")

I believe that in both cases ${PipeErrors[0]} reflects the error status of ProduceCommand. Further, it would be logical to assume that ${PipeErrors[1]} reflects the error status of tee or pee itself, respectively.

But this leads me into at least two understanding problems:

  1. What is the error status (return value) of tee or pee? I didn't find precise statements about that in the man pages. Do they return a hard-coded error status if one of the consume commands fails, or do they relay the error status of the consume commands somehow (as ssh does, for example)? If the former is the case, how can we find out which of the consume commands is the culprit? If the latter is the case, which error status is relayed? Is it simply that of the command which fails first?

  2. AFAIK, bash or the tee or pee command itself, respectively, internally use pipes (fifos) to get ProduceCommand's output to the consume commands. This means that we have a pipe whose (first and in this case, only) receiving side is a pipe itself. This should not influence $PipeErrors in the sample code above, but I am really unsure.

Could somebody shed some light on this?

2 Answers 2

1

What is the error status (return value) of tee

It's 0 when it was able to copy all the data to all the output files, and >0 if not. See the spec. The GNU coreutils implementation of tee has extra options to ignore errors when writing to pipes (as those used to implement >(...)):

$ seq 1024 | tee >(false) >/dev/null; echo $?
141
$ seq 1024 | tee -p >(false) >/dev/null; echo $?
0

There are no options to know which of the outputs has failed, if any [1].


But your question seems to be rather about whether the exit status of the commands run in >(..) process substitutions is reflected in any way in PIPESTATUS, or if it could be in any way reflected in PIPESTATUS.

The answer to that is No.

First of all, notice that >(...) are more like ... & background commands than like ...|... pipeline commands. In a snippet like:

... | tee >(cmd ...) | ...; echo ${PIPESTATUS[@]}

there's no guarantee that cmd has finished at all by the time you run echo ${PIPESTATUS[@]}.

But they're not exactly like ...& either as you cannot wait for them and cannot obtain their status from $! -- except in some limited cases which do not include their use with tee or other external commands:

$ bash -c 'echo 1 | tee >(sleep 2; sed s/1/2/); wait; echo DONE'
1
DONE
$
<after two seconds>
2

As you see, both tee and the main shell have finished long before the command from >(...).

[1] A command like pee which is running the output "subcommands" itself (and is waiting for them to finish) could be smarter and reflect in its exit status which of them have failed (e.g. by setting bit 1 for the first, bit 2 for the second, and so on, up to 8 subcommands), but does no such thing either.

1
  • Thank you very much, accepted and +1. That was enlightening. I guess I then can use neither tee nor pee for my purposes. Even either one of the two disadvantages (background execution, not being able to find out which of the outputs has failed) on its own would prevent that. Thanks also for the link to the spec; too bad that this information is not in the man page. Out of curiosity, I'll check whether it is in the info page at least.
    – Binarus
    Commented Oct 8, 2021 at 5:53
1

You can always do:

{
  {
    ProduceCommand 2>/dev/null 3>&- ||
      HandleErrorInProduceCommand >&3 3>&-
  } |
    tee >(
      ConsumeCommand1 3>&- ||
        HandleErrorInConsumer1 >&3 3>&-
    ) >(
      ConsumeCommand2 3>&- ||
        HandleErrorInConsumer2 >&3 3>&-
    ) > /dev/null
} 3>&1

That handles the error in each of the subshells where the producer and consumers are started.

We duplicate stdout onto fd 3, so can restore the original stdout for the error handlers as we don't want their output if any to go through the pipes.

If you'd rather the error handlers be run within the main shell process (so it can exit it for instance), you could get those subshells to pass the exit status to the parent shell via some command substitution pipe:

 producer_status=-1
consumer1_status=-1
consumer2_status=-1
{
  eval "$(
    {
      {
        ProduceCommand 2>/dev/null 4>&-
        echo "producer_status=$?" >&4
      } | tee >(
        ConsumeCommand1 4>&-
        echo "consumer1_status=$?" >&4
      ) >(
        ConsumeCommand2 4>&-
        echo "consumer2_status=$?" >&4
      )
    } 4>&1 >&3 3>&-
  )"
} 3>&1

[ "$producer_status"  -eq 0 ] || HandleErrorInProduceCommand
[ "$consumer1_status" -eq 0 ] || HandleErrorInConsumer1
[ "$consumer2_status" -eq 0 ] || HandleErrorInConsumer2

That avoids the $PIPESTATUS bashism or you could avoid the >(...) kshism and replace them with a normal pipeline:

{
  ProduceCommand 2>/dev/null |
    {
      tee /dev/fd/4 |
        ConsumeCommand1 4>&-
    } 4>&1 >&3 3>&- |
      ConsumeCommand2 3>&-
} 3>&1
 producer_status=${PIPESTATUS[0]}
consumer1_status=${PIPESTATUS[1]}
consumer2_status=${PIPESTATUS[2]}

[ "$producer_status"  -eq 0 ] || HandleErrorInProduceCommand
[ "$consumer1_status" -eq 0 ] || HandleErrorInConsumer1
[ "$consumer2_status" -eq 0 ] || HandleErrorInConsumer2

Or combine the two approaches and then get standard sh syntax and also get access to tee's exit status as a bonus.

 producer_status=-1
      tee_status=-1
consumer1_status=-1
consumer2_status=-1

{
  eval "$(
    {
      {
        ProduceCommand 2>/dev/null 4>&-
        echo "producer_status=$?" >&4
      } 3>&- |
        {
          {
            tee /dev/fd/5 4>&-
            echo "tee_status=$?" >&4
          } |
            ConsumeCommand1 4>&-
          echo "consumer1_status=$?" >&4
        } 5>&1 >&3 3>&- |
        ConsumeCommand2 >&3 3>&- 4>&- 
        echo "consumer2_status=$?" >&4
    } 4>&1
  )"
} 3>&1

[ "$producer_status"  -eq 0 ] || HandleErrorInProduceCommand
[ "$tee_status"       -eq 0 ] || HandleErrorInTee
[ "$consumer1_status" -eq 0 ] || HandleErrorInConsumer1
[ "$consumer2_status" -eq 0 ] || HandleErrorInConsumer2

Note that tee may die of a SIGPIPE if one of the processes exits without reading all the input, which means the other process may also miss some input. So it may be important to also check its exit status.

As already noted by @UncleBilly, with the GNU implementation of tee, that can be worked around with the -p option (where tee ignores SIGPIPE signals and just stops trying to write more data to a pipe when it becomes broken).

With other implementations, you can replace tee ... with (trap '' PIPE; exec tee ...) to get a similar behaviour (though possibly some error messages about the broken pipes even if tee is not aborted).

3
  • Thank you very much for this in-depth answer - +1! But TBH, even after looking at it for an hour, I haven't understood it yet. But I believe I'll be able to understand the answer once I've understood the first example. Therefore I hope you won't mind me asking some questions about that first example (one per comment for clarity). First question: In the process substitutions, shouldn't the pipe symbol (|) rather be a shortcut-or (||)?
    – Binarus
    Commented Oct 8, 2021 at 16:11
  • 1
    @Binarus, d'oh. Yes, sorry, those were meant to be ||, | would not make sense. Fixed now. Commented Oct 8, 2021 at 16:14
  • I still had some problems with understanding the code, especially the first example, but I've got it now (see unix.stackexchange.com/questions/672750). Thanks again! By the way, I like PIPESTATUS, and I don't like eval very much, so your third example probably shows the right way (for me) :-)
    – Binarus
    Commented Oct 11, 2021 at 17:07

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .