2

Please consider the following log from a terminal session (Debian Buster, Bash 5.0):

root@cerberus ~/scripts # rm -f result
root@cerberus ~/scripts # { { echo test; } | cat > result; }
root@cerberus ~/scripts # cat result
test
root@cerberus ~/scripts #

Nothing special here, this is the expected behavior, and I understand it.

But I do not understand the behavior in the following case:

root@cerberus ~/scripts # rm -f result
root@cerberus ~/scripts # { { echo test >&3; } | cat > result; } 3>&1
test
root@cerberus ~/scripts # cat result
root@cerberus ~/scripts #

To be precise, I believe that I understand why "test" is output when executing the second line, but I don't understand why nothing goes into the result file. My understanding of what happens is the following:

  1. At first, fd 3 is set up as a duplicate of stdout. I am sure that this happens before the pipe is executed, because otherwise none of the commands in the pipe would have access to fd 3 at all, which would cause a "bad descriptor" error message.

  2. A pipeline is not a simple command, so a subshell is spawn to execute it. The subshell inherits the parent shell's execution environment, including file descriptors and redirections. [1]

  3. Each of the commands in the pipeline also is executed in its own subshell [2], again inheriting the execution environment and the file descriptors. echo's output is redirected to fd 3, which in turn had been duplicated from stdout before, which in summary leads echo's output to appear on stdout (output goes to fd 3, which goes to fd 1, which is stdout).

  4. But I don't get why echo's output doesn't make it into the result file. From the bash manual (emphasis mine):

The output of each command in the pipeline is connected via a pipe to the input of the next command. That is, each command reads the previous command’s output. This connection is performed before any redirections specified by the command.

I am understanding this in the sense that echo's output should be connected to cat's input before the redirection >&3 is set up or applied, respectively. But if that would be true, the result file would exist (and contain "test") after the command has been executed. Therefore my understanding is obviously wrong.

Could somebody please explain what I am missing?

Update, based on A.B's and Gilles' excellent answer below, with further explanations

The source of my worries is what I wrote in item 3. above. It just does not work that way; see also Gilles' answer.

A.B was the first to provide an answer (see below). However, I needed some time to understand it. Therefore I'll explain some passages so that they can be understood more easily.

  1. Last part of the line: 3>&1 is done first: fd 1 pointing to the terminal output is duplicated to fd 3. This means that fd 1 and fd 3 now both point to the terminal output. They are identical and can be used interchangeably.

  2. Before forking, a pipe is created, typically using the pipe(2) system call, on next available fds: let's say fd 4 and fd 5. The preparation process then forks into future echo and future cat, performing the following steps:

    a) The preparation process for echo works like that:

    fd 5 is duplicated to fd 1 (overwriting where fd 1 pointed to: the terminal output). That means that fd 1 is now identical to fd 5, and that they can be used interchangeably. Specifically, fd 1 does not point to the terminal output any more, but points to the writing end of the pipe.

    At this stage (but see below), the output of echo would go to the writing end of the pipe, because echo writes to fd 1, which points to that writing end.

    Because we don't need two file descriptors for the same thing, and because echo writes to fd 1 anyway, fd 5 gets closed now.

    Then echo is executed, but after having set up the additional redirection noted behind it (see 3.).

    b) Likewise, the preparation process for cat duplicates fd 4 to fd 0, meaning that fd 0 no longer points to the terminal input, but points to the receiving side of the pipe. At this stage, the input for cat would come from the receiving side of the pipe, because cat reads from fd 0, and fd 0 is connected to that receiving side. Because we don't need two file descriptors for the same thing, and because cat reads from fd 0 anyway, fd 4 gets closed now. Then cat is executed.

    While this all happens, fd 3 is inherited everywhere.

  3. >&3 does the opposite of bullet 1: It duplicates fd 3 to fd 1. fd 3 had been created so that it points to the terminal output, and is inherited by the subshell which executes the pipe and the further subshells which execute the individual pipe commands.

    In step 2a), fd 1 had been pointed to the writing side of the pipe. But now, the redirection >&3 overwrites fd 1 again and makes it equal to fd 3, which in turn (still) points to the terminal output. This means that fd 1 no longer points to the writing side of the pipe, but to the terminal output instead. This is the reason why "test" appears on the terminal when the pipe is executed (remember that echo always writes to fd 1, regardless of where fd 1 points to).

    Plus, when fd 1 gets "overwritten" by the redirection, its old version gets closed (because the underlying system call dup2(2) does that). Since its old version was pointing to the writing end of the pipe, that writing end is now closed.

    Because of this, the receiving end, and thus, cat, won't receive any data. They immediately get an EOF notification instead. This is the reason why cat does not receive anything and why consequently the result file remains empty, or is truncated.

    [ Side note: I should have closed fd 3 after the redirection (that is, we should have written >&3 3>&- instead of >&3), because echo -as mentioned above- writes to fd 1 and does not know anything about fd 3 at all. However, that part was missing in my example, and I'd like to leave it that way to not distract from the actual problem). ]

7
  • fd3 is not a duplicate of stdout; everything that would have gone to stdout is sent to fd3 instead -- it's a redirection, not a copy. There is now nothing sent to stdout, which is why result is empty (> redirects stdout). At the end, fd3 gets redirected to stdout (3>&1), but this is after the redirection to result. Oct 11, 2021 at 12:02
  • The pipe only forwards stdout, so cat only sees stdout, which is empty (the actual stdout has been redirected to fd3). fd3 is able to be seen by the 3>&1 because of the outer set of braces; if you remove that outer set of braces, then the command fails because fd3 is only available to cat, not to cat's output. Oct 11, 2021 at 12:10
  • @JeffBreadner Thank you very much. But you said: "fd3 is not a duplicate of stdout; ..." But the manual says: "The operator [n]>&word is used similarly to duplicate output file descriptors." (gnu.org/savannah-checkouts/gnu/bash/manual/…) (it's even in the URL :-)).
    – Binarus
    Oct 11, 2021 at 13:48
  • @zevzek Thank you very much. But no, I never used anything else than bash. Regarding your last sentence in your first comment: Because that's the way how I've understood the snippet from the manual I have cited, which was making me believe that the connection stdout->stdin is established first and a later redirection does not destroy it again. Otherwise, the sentence in bold could be left away at all, couldn't it?
    – Binarus
    Oct 11, 2021 at 13:56
  • @Binarus The documentation may say that, but compare what is actually happening with both its description and mine, and you will see that it is redirecting the output, not copying it. Or, consider this simple example: echo test versus echo test > /some/file. If it was simply cloning the output, then the 2nd command would also print 'test' to the console, the same as the first, but it does not. Oct 11, 2021 at 14:22

2 Answers 2

1

That's because of OP's bullet 4 that it works like this, with fd inherited along the various creation/exec of processes. I'm not writting all places where fork/exec happens. I'm certainly simplifying some of it (what with built-in commands...). Documentation links provided for Linux, but the same behavior is supposed to happen on any POSIX or POSIX-like system.

  1. last part of the line: 3>&1 is done first: fd 1 pointing to the terminal is duplicated as fd 3 (typically using the dup2(2) system call).
  2. Before forking, a pipe is created, typically using the pipe(2) system call, on next available fds: let's say 4 and 5. The preparation process then forks into future echo and future cat. proto-echo dups 5 to 1 ("overwritting" where it pointed to: the terminal), closes 5 and execs echo, proto-cat dups2() 4 to 0, closes 4 and execs cat. fd 3 is inherited everywhere.
  3. >&3 does the opposite of bullet 1: it duplicates fd 3 (pointing to the terminal) to fd 1. So the writing side of the pipe was replaced and is now closed (dup2(2) says: "If the file descriptor newfd was previously open, it is silently closed before being reused."). Nothing will ever be written to the pipe. Terminal receives test and displays it.
  4. in parallel cat opens and truncates destination file result and start reading from the pipe. This triggers an EOF as per pipe(7) because the write side is/was closed: cat command ends.
  5. main shell process has no child remaining: execution ends

Result: test on terminal and empty result file.

4
  • Thank you very much, accepted and +1. The key to understanding was your paragraph 2. Too bad that in the normal docs it is not mentioned that bash uses the pipe(2) system call, and that it duplicates the new fds to fd1 and fd0, respectively. Did you read bash's source code to find that information? It is neither in the POSIX docs nor in man bash (although it admittedly is logical that it works that way).
    – Binarus
    Oct 11, 2021 at 15:27
  • 1
    That's the standard method for communicating through pipes between a parent process and a child process. It's not specific to bash, but to using pipe(2). I think at the end of pipe(2)'s manual there is a complete C example.
    – A.B
    Oct 11, 2021 at 16:05
  • Thank you very much again. I've seen the example, but it doesn't show what was the most important step in understanding (for me). That is, it does not duplicate fd 5 to fd 1 and fd 4 to fd 0. Instead, the parent process in the example writes to fd 5, and the child process reads from fd 4. Of course, I am aware that the duplications are just not needed in that example, but for me they were the crucial step in understanding what bash (and probably other shells) do.
    – Binarus
    Oct 11, 2021 at 16:37
  • Hem, but 4 and 5 are closed everywhere once their ephemeral role was played. echo's fd 1 got the write side of the pipe from 5 and cat's fd0 got the read side of the pipe from 4. (then echo (or yet still proto-echo?) got its fd 1 replaced).
    – A.B
    Oct 11, 2021 at 17:40
1

1. At first, fd 3 is set up as a duplicate of stdout.

That's correct as stated, but a bit weirdly stated, and you seem to have misunderstood what this means. It does not mean that where this redirection is in effect, writing to fd 3 is equivalent to writing to stdout. It means that fd 3 is connected to whatever stdout is connected to at the point where the redirection is set up. If you're running this code in a terminal, 3>&1 connects file descriptor 3 to the terminal. And so…

3. (…) echo's output is redirected to fd 3, which in turn had been duplicated from stdout before, which in summary leads echo's output to appear on stdout (output goes to fd 3, which goes to fd 1, which is stdout).

FD 3 is the terminal. The fact that at some point it happened to also be fd 1 of some other process is an irrelevant historical detail.

1
  • Thank you very much for the answer, upvoted. Yes, you have exactly described my misunderstanding and the source of my worries. I needed some time to follow the first answer, but finally got it and wrote the update / edit to my question.
    – Binarus
    Oct 13, 2021 at 21:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.