Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. It only takes a minute to sign up.
Sign up to join this community
Does anything symbolic happen in chaining bash commands via a pipe or is it all compute-pass-compute-pass?
For example in head t.txt -n 5 | tail -n 2, is head t.txt -n 5 getting computed and then tail -n 2 executes over it. Or first there is some abstraction to tell the shell that lines 3 to 5 are to be read? It might not make a difference in this example, but I guess can in other scenarios.
The shell uses the pipe(2) system call to create a bounded buffer in the kernel with two file descriptors, one to enable processes to write to the buffer, and another to enable processes to read from the buffer.
Consider a simple case:
$ p1 | p2
In this case, conceptually, the shell creates the above-mentioned pipe, fork()s, the child connects its standard output stream to the write-end of the pipe, then the child exec()s p1. Next, the shell fork()s again, the child connects its standard input stream to the read-end of the pipe, then the child exec()s p2. (I say conceptually because shells might do things in different orders, but the idea is the same.)
At that point, p1 and p2 are running concurrently. p1 will write to the pipe, and the kernel will copy the written data to the buffer. p2 will read from the pipe, and the kernel will copy the read data from the buffer. If the pipe gets full, then the kernel will block p1 in its call to write() until p2 reads something from the pipe, freeing up some space. If the pipe is empty, then the kernel will block p2 in its call to read() until p1 writes more data to the pipe.
Of the two models that you suggest. compute-pass-compute-pass is the closest. The shell just connects the processes. It knows nothing of what they are doing.
Except, The order of execution is undefined. They effectively run at the same time. However the one on the left must output bytes before the one on the right inputs them. Data flows left to right. Data flows from the first command, out of its standard out, it then flows into the standard in, of the next process, where it is processed, it then comes out of its standard out, where it can be piped to another process, etc, etc, etc.
If there is no redirection >, <, etc. or reading from a file. Then it looks like this.
┌───────────┐ ┌───────────┐ ┌─────────────┐
Terminal⇨│Process one│⇨│Process two│⇨│Process Three│⇨Terminal
└───────────┘ └───────────┘ └─────────────┘
ps aux | grep initwhengreplists itself.a | bis basicallya > tmpfile; b < tmpfile. That's WRONG: a command likeyes | head -n20would never finish if pipes worked like that. So yes, something highly symbolic happens in a pipeline ;-)