How Bash built-in control structure / command redirection works

Question

I'm interested how redirection is technically implemented with bash built-in control structures and functions.

For example, I have the following command

while read line; do echo $line; done < lines.txt | tac > ~/reversed.txt

What mechanism connects the stdin (lines.txt) to read command (argument of while) and what connects do body to stdout (pipe)? There are clearly some contextual rules applied (contra external command redirection), but what exactly are those rules and how bash technically implements them?

Answer 1

Usually, UNIX shells just open the required files with open, then fork themselves, then dup2 the previously obtained fd's to stdin/stdout/stderr (0/1/2) so they're handled accordingly by the program they execve later. It might be different with builtin commands in order to improve performance (as fork and execve are quite expensive), but the semantics is the same.

If you mean the command line parsing rules, though, it's described in POSIX. They don't discern builtins and external programs.

Answer 2

Redirecting file descriptor d0 to or from a file involves the following steps:

1. Open the file. The file is opened on some file descriptor d1.
2. Duplicate the descriptor d0 to a currently-unused file descriptor d2 which is larger than d0. This can be done with the F_DUPFD command of the fcntl system call. If d0 is not open then do nothing for this step.
3. Duplicate d1 to d0. This can be done with F_DUPFD or with dup2.
4. Close d1.

The reason the duplication shuffle is needed is that applications don't get to choose the file descriptor when opening the file. Steps 2–4 can be omitted if d1 = d0 but the shell can't guarantee this.

When the redirection applies to an external command, it is performed in the child process, after the child process has been created with fork but before executing the external command with execve. When the redirection applies to an internal shell command (e.g. a function call, a loop, etc.), these steps have to be performed in the original process¹, and the shell needs to restore the original file descriptor state after the redirected command completes, by duplicating d2 back to d0 and closing d2 (or just closing d0 if it wasn't initially open).

A pipe involves similar steps, but it is a bit more complicated because creating the pipe creates two file descriptors (the read end and the write end) and there are two subprocesses.

1. Create a pipe with pipe. The pipe system call returns a pair of file descriptors {r, w}.

2. In the left-hand side of the pipe:

   1. Close r.
   2. Do the duplication shuffle to move w to 1.

3. In the right-hand side of the pipe:

   1. Close w.
   2. Do the duplication shuffle to move r to 0.

4. In shells that execute both sides of the pipe in subprocesses, the parent process closes r and w, then waits for the two sides of the pipe to terminate.

   In shells that execute the right-hand side of the pipe in the parent process, the shell waits for the left-hand side to terminate, then closes 0 and restores the original file descriptor at 0.

You can look at what shells do by reading their source code, or by following them in operation with a debugger. For example, on Linux, look the system calls in action with

strace sh -c '…'

¹ _{Ancient shells (before POSIX) executed redirected complex commands in a subprocess, so they didn't require any redirection restoration.}