Stderr of piped shell scripts isn't always displayed

Question

I piped my own shell scripts for some testing and accidentally noticed something strange. Namely, the stderr of those piped processes isn't always displayed on the screen.

I simplified the scripts and here's my session with bash:

$ cat file1
echo stdout
echo stderr 1>&2
$ cat file2
echo stdout2
echo stderr2 1>&2
$ cat file3
echo stdout3
echo stderr3 1>&2

$ ./file1 | ./file2 | ./file3
stderr2
stderr
stdout3
stderr3

$ ./file1 | ./file2 | ./file3
stderr
stdout3
stderr3

$ ./file1 | ./file2 | ./file3
stderr2
stdout3
stderr3

I know that the stdout of the first two scripts is lost, since file3 doesn't read anything, but it doesn't matter here.

What's happening with the stderr2 and stderr outputs in the 2nd and 3rd case, respectively?

Answer 1

Ooh, a race condition! There are a few things going on here behind the scenes to produce this effect.

First, if a process tries to write to a pipe whose other end has been closed, that process receives a SIGPIPE signal. By default, this ends the process. Why would you want this? If you run cat my_huge_file | head -3, you'll only see the first three lines of the file, and because of the signaling mechanism, cat knows to stop after those three lines. Without it, it would read the whole file.

Second, all processes in a pipeline run in parallel, and which process will finish first is unpredictable. When a process ends, all of its file descriptors are closed -- including the read ends of any pipes.

So what's happening is that sometimes, for example, file3 finishes completely before file2 has written to its stdout. Then, when file2 does try to write, it gets SIGPIPE, and quits without reaching the echo stderr2 line. But then other times, file2 manages to write to stdout before file3 completes, and in this case it gets to continue.