123

I've never really thought about how the shell actually executes piped commands. I've always been told that the "stdout of one program gets piped into the stdin of another," as a way of thinking about pipes. So naturally, I thought that in the case of say, A | B, A would run first, then B gets the stdout of A, and uses the stdout of A as its input.

But I've noticed that when people search for a particular process in ps, they'd include grep -v "grep" at the end of the command to make sure that grep doesn't appear in the final output.
This means that in the command ps aux | grep "bash" | grep -v "grep" it is implied that ps knew that grep was running and therefore is in the output of ps. But if ps finishes running before its output gets piped to grep, how did it know that grep was running?

flamingtoast@FTOAST-UBUNTU: ~$ ps | grep ".*"
PID TTY          TIME CMD
3773 pts/0    00:00:00 bash
3784 pts/0    00:00:00 ps
3785 pts/0    00:00:00 grep
1
  • why not accept an answer? Commented Aug 21, 2019 at 13:21

5 Answers 5

94

Piped commands run concurrently. When you run ps | grep …, it's the luck of the draw (or a matter of details of the workings of the shell combined with scheduler fine-tuning deep in the bowels of the kernel) as to whether ps or grep starts first, and in any case they continue to execute concurrently.

This is very commonly used to allow the second program to process data as it comes out from the first program, before the first program has completed its operation. For example

grep pattern very-large-file | tr a-z A-Z

begins to display the matching lines in uppercase even before grep has finished traversing the large file.

grep pattern very-large-file | head -n 1

displays the first matching line, and may stop processing well before grep has finished reading its input file.

If you read somewhere that piped programs run in sequence, flee this document. Piped programs run concurrently and always have.

4
  • 14
    And what's cool about this example is that when head gets the one line it needs, it terminates and when grep notices this, it also terminates without doing a bunch of further work for nothing.
    – Joe
    Commented May 5, 2012 at 20:48
  • I guess there is some kind of IO buffer concerning the pipe... how do I know it's size in bytes? What do I want to read to learn more about it? :)
    – n611x007
    Commented Nov 24, 2013 at 21:14
  • 4
    @naxa There are two buffers, actually. There's the stdio buffer inside the grep program, and there's a buffer managed by the kernel in the pipe itself. For the latter, see How big is the pipe buffer? Commented Nov 24, 2013 at 21:29
  • The grep option --line-buffered will keep grep from buffering. Thus, grep --line-buffered First_Line very-big-file | ...` will produce its output at once, not waiting for an output bufferfull.
    – waltinator
    Commented Sep 30, 2023 at 2:37
54

The order the commands are run actually doesn't matter and isn't guaranteed. Leaving aside the arcane details of pipe(), fork(), dup() and execve(), the shell first creates the pipe, the conduit for the data that will flow between the processes, and then creates the processes with the ends of the pipe connected to them. The first process that is run may block waiting for input from the second process, or block waiting for the second process to start reading data from the pipe. These waits can be arbitrarily long and don't matter. Whichever order the processes are run, the data eventually gets transferred and everything works.

5
  • 6
    Nice answer, but the OP seems to think the processes run sequentially. You might make it clearer here that the processes are run concurrently, and the pipe is like.... a pipe between buckets, where water flows through all at the (approx.) same time.
    – Keith
    Commented Apr 28, 2012 at 6:16
  • 1
    Thank you for the clarification. The sources I've been reading made it seem like piped programs ran sequentially, rather than concurrently. Commented Apr 28, 2012 at 7:58
  • 1
    To see experience the processes starting in an undetermined fashion try running this 1000 times: echo -n a >&2 | echo b >&2
    – Ole Tange
    Commented May 4, 2016 at 7:43
  • Maybe a naive question...Has the first process being blocked/waiting on the second process (for a piece information) anything to do w/ the pipe? But then pipes must be bidirectional?
    – stdout
    Commented Sep 26, 2020 at 15:38
  • 1
    @stdout If the writing process is blocked it is because the pipe is full and the reading process needs to read some data out of the pipe so that there is space. If the reading process is blocked it is because the pipe is empty and the writing process needs to write some data into the pipe. The blocking happens because either the pipe is empty or the pipe is full (pipes have finite size of course).
    – Kyle Jones
    Commented Sep 26, 2020 at 15:52
36

At the risk of beating a dead horse, the misconception seems to be that

    A | B

is equivalent to

    A > temporary_file
    B < temporary_file
    rm temporary_file

But, back when Unix was created and children rode dinosaurs to school, disks were very small, and it was common for a rather benign command to consume all the free space in a file system.  If B was something like grep some_very_obscure_string, the final output of the pipeline could be much smaller than that intermediate file.  Therefore, the pipe was developed, not as a shorthand for the “run A first, and then run B with input from A’s output” model, but as a way for B to execute concurrently with A and eliminate the need for storing the intermediate file on disk.

0
1

Typically you run this under bash. process working and starting concurrently, but are running by the shell in parallel. How is it possible?

  1. if it isn't last command in pipe, create unnamed pipe with pair of sockets
  2. fork
  3. in child reassign stdin/stdout to sockets if it's needed (for first process in pipe stdin is not reassigned, the same for last process and his stdout)
  4. in child EXEC specified command with arguments that sweep out original shell code, but leaves all opened by them sockets. child process ID will not be changed because this is the same child process
  5. concurrently with child but parallel under main shell go to step 1.

system not guarantee how fast exec will be executed and specified command starts. it's independent to the shell, but system. This is because:

ps auxww| grep ps | cat

once show grep and/or ps command, and next now. It depends how fast kernel really start processes using system exec function.

1
  • 1
    Concurrent execution means that two or more processes execute within the same time frame, usually with some sort of dependency between them. Parallel execution means that two or more processes execute simultaneously (e.g. on separate CPU cores at the same time). Parallelism is not relevant to the question, nor is "how fast" exec() is executed, but how the exec() calls and execution of the programs in a pipe are interleaved. Commented Sep 6, 2013 at 10:25
-2

You asked about the order and I think that this is a very important aspect to the matter. This is not random (as Gilles tries to say in his answer).

Here is a ps -ef piped to the grep command:

$ ps -ef | grep .
...
alexis   37188 55443  0 20:17 pts/4    00:00:00 ps -ef
alexis   37189 55443  0 20:17 pts/4    00:00:00 grep --color=auto .
...

Note: I removed all the other processes from the output since they are of no importance to the matter.

As we can see, there is a ps -ef and a grep --color=auto . in the output. Can you answer your question now?

Yes. The ps command has PID 37,188 and the grep command has PID 37,189. Clearly, they were created left to right and no shell should do this differently.

Technically, in C, we create pipes with the pipe(2) function which gives us two file descriptors. One is going to be used as the stdout of ps and the other as the stdin of grep. It is easy enough to hold on the file descriptor for stdin until after you started ps.

Further, if you look at your system configuration like so:

$ getconf -a | grep PIPE_BUF
PIPE_BUF                           4096
_POSIX_PIPE_BUF                    4096

you notice those two parameters defining the minimum guaranteed size of the pipe in bytes. Since Linux 2.6, the default size is 64Kb. Also the absolute maximum number of bytes is defined in:

$ cat /proc/sys/fs/pipe-max-size
1048576

and we can see this is 1Mb. Once the pipe is full, the outputter (ps in our first example) blocks until data gets read by the process on the other side of the pipe (grep in our first example).

In other words, since the output of ps is much less than the size of a pipe:

$ ps -ef | wc
   1132   10819  121435

(i.e. about 120Kb of output on my computer at the moment...)

The pipeline won't get blocked at all.

For streamed data over 1Mb, it does get blocked at some point. If grep was not started immediately, it would never start since the write() call in the first command would then be blocked.

So the processes are very quickly started back to back, but for most of the time, they run in parallel (or concurrently if you have a single processor). That is, the ps command will die first. This marks the pipe as "done" (you get the EOF signal when reading data from it) and that's how the next tool knows it is done and it also dies once it processes the last few bytes it received.

Conversely, if a process on the right side of a pipe dies early (before the one of the left is done writing to the pipe), then the process on the left receives the SIGPIPE signal as soon as it tries to write to the pipe. This is done like so to make sure that the pipeline dies quickly if any of the processes within it dies.

1
  • 1
    No, PIPE_BUF is the maximum amount that can be guaranteed to be written atomically which is much less than the pipe size. Since Linux 2.6, the pipe size is by default 64 kB and can be increased up to the maximum as configured in /proc/sys/fs/pipe-max-size which is 1 MB by default.
    – Juergen
    Commented Sep 28, 2023 at 15:51

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .