I've never really thought about how the shell actually executes piped commands. I've always been told that the "stdout of one program gets piped into the stdin of another," as a way of thinking about pipes. So naturally, I thought that in the case of say, A | B, A would run first, then B gets the stdout of A, and uses the stdout of A as its input.

But I've noticed that when people search for a particular process in ps, they'd include grep -v "grep" at the end of the command to make sure that grep doesn't appear in the final output.
This means that in the command ps aux | grep "bash" | grep -v "grep" it is implied that ps knew that grep was running and therefore is in the output of ps. But if ps finishes running before its output gets piped to grep, how did it know that grep was running?

flamingtoast@FTOAST-UBUNTU: ~$ ps | grep ".*"
3773 pts/0    00:00:00 bash
3784 pts/0    00:00:00 ps
3785 pts/0    00:00:00 grep
  • why not accept an answer? Aug 21 '19 at 13:21

Piped commands run concurrently. When you run ps | grep …, it's the luck of the draw (or a matter of details of the workings of the shell combined with scheduler fine-tuning deep in the bowels of the kernel) as to whether ps or grep starts first, and in any case they continue to execute concurrently.

This is very commonly used to allow the second program to process data as it comes out from the first program, before the first program has completed its operation. For example

grep pattern very-large-file | tr a-z A-Z

begins to display the matching lines in uppercase even before grep has finished traversing the large file.

grep pattern very-large-file | head -n 1

displays the first matching line, and may stop processing well before grep has finished reading its input file.

If you read somewhere that piped programs run in sequence, flee this document. Piped programs run concurrently and always have.

  • 10
    And what's cool about this example is that when head gets the one line it needs, it terminates and when grep notices this, it also terminates without doing a bunch of further work for nothing.
    – Joe
    May 5 '12 at 20:48
  • I guess there is some kind of IO buffer concerning the pipe... how do I know it's size in bytes? What do I want to read to learn more about it? :)
    – n611x007
    Nov 24 '13 at 21:14
  • 4
    @naxa There are two buffers, actually. There's the stdio buffer inside the grep program, and there's a buffer managed by the kernel in the pipe itself. For the latter, see How big is the pipe buffer? Nov 24 '13 at 21:29

The order the commands are run actually doesn't matter and isn't guaranteed. Leaving aside the arcane details of pipe(), fork(), dup() and execve(), the shell first creates the pipe, the conduit for the data that will flow between the processes, and then creates the processes with the ends of the pipe connected to them. The first process that is run may block waiting for input from the second process, or block waiting for the second process to start reading data from the pipe. These waits can be arbitrarily long and don't matter. Whichever order the processes are run, the data eventually gets transferred and everything works.

  • 5
    Nice answer, but the OP seems to think the processes run sequentially. You might make it clearer here that the processes are run concurrently, and the pipe is like.... a pipe between buckets, where water flows through all at the (approx.) same time.
    – Keith
    Apr 28 '12 at 6:16
  • Thank you for the clarification. The sources I've been reading made it seem like piped programs ran sequentially, rather than concurrently. Apr 28 '12 at 7:58
  • 1
    To see experience the processes starting in an undetermined fashion try running this 1000 times: echo -n a >&2 | echo b >&2
    – Ole Tange
    May 4 '16 at 7:43
  • Maybe a naive question...Has the first process being blocked/waiting on the second process (for a piece information) anything to do w/ the pipe? But then pipes must be bidirectional?
    – stdout
    Sep 26 '20 at 15:38
  • 1
    @stdout If the writing process is blocked it is because the pipe is full and the reading process needs to read some data out of the pipe so that there is space. If the reading process is blocked it is because the pipe is empty and the writing process needs to write some data into the pipe. The blocking happens because either the pipe is empty or the pipe is full (pipes have finite size of course).
    – Kyle Jones
    Sep 26 '20 at 15:52

At the risk of beating a dead horse, the misconception seems to be that

    A | B

is equivalent to

    A > temporary_file
    B < temporary_file
    rm temporary_file

But, back when Unix was created and children rode dinosaurs to school, disks were very small, and it was common for a rather benign command to consume all the free space in a file system.  If B was something like grep some_very_obscure_string, the final output of the pipeline could be much smaller than that intermediate file.  Therefore, the pipe was developed, not as a shorthand for the “run A first, and then run B with input from A’s output” model, but as a way for B to execute concurrently with A and eliminate the need for storing the intermediate file on disk.


Typically you run this under bash. process working and starting concurrently, but are running by the shell in parallel. How is it possible?

  1. if it isn't last command in pipe, create unnamed pipe with pair of sockets
  2. fork
  3. in child reassign stdin/stdout to sockets if it's needed (for first process in pipe stdin is not reassigned, the same for last process and his stdout)
  4. in child EXEC specified command with arguments that sweep out original shell code, but leaves all opened by them sockets. child process ID will not be changed because this is the same child process
  5. concurrently with child but parallel under main shell go to step 1.

system not guarantee how fast exec will be executed and specified command starts. it's independent to the shell, but system. This is because:

ps auxww| grep ps | cat

once show grep and/or ps command, and next now. It depends how fast kernel really start processes using system exec function.

  • 1
    Concurrent execution means that two or more processes execute within the same time frame, usually with some sort of dependency between them. Parallel execution means that two or more processes execute simultaneously (e.g. on separate CPU cores at the same time). Parallelism is not relevant to the question, nor is "how fast" exec() is executed, but how the exec() calls and execution of the programs in a pipe are interleaved. Sep 6 '13 at 10:25

You asked about the order and I think that this is a very important aspect to the matter. This is not random (as Gilles tries to say in his answer).

Here is a ps -ef piped to the grep command:

$ ps -ef | grep .
alexis   37188 55443  0 20:17 pts/4    00:00:00 ps -ef
alexis   37189 55443  0 20:17 pts/4    00:00:00 grep --color=auto .

Note: I removed all the other processes from the output since they are of no importance to the matter.

As we can see, there is a ps -ef and a grep --color=auto . in the output. Can you answer your question now?

Yes. The ps command has PID 37,188 and the grep command has PID 37,189. Clearly, they were created left to right and no shell should do this differently.

Technically, in C, we create pipes with the pipe(2) function which gives us two file descriptors. One is going to be used as the stdout of ps and the other as the stdin of grep. It is easy enough to hold on the file descriptor for stdin until after you started ps.

Further, if you look at your system configuration like so:

$ getconf -a | grep PIPE_BUF
PIPE_BUF                           4096
_POSIX_PIPE_BUF                    4096

you notice those two parameters which define the size of the pipe in bytes. This is the maximum number of bytes that can be in the pipe. Once the pipe is full, the outputter (ps in our first example) blocks until data gets read by the process on the other side of the pipe (grep in our first example).

In other words, since the output of ps is much more than the size of a pipe:

$ ps -ef | wc
   1132   10819  121435

(i.e. about 120Kb of output on my computer at the moment...)

The pipeline would block very quickly if all the output of ps was necessary before grep was started.

So the processes are very quickly started back to back, but for most of the time, they run in parallel (or concurrently if you have a single processor). That is, the ps command will die first. This marks the pipe as "done" (you get the EOF signal when reading data from it) and that's how the next tool knows it is done and it also dies once it processes the last few bytes it received.

Conversely, if a process on the right side of a pipe dies early (before the one of the left is done writing to the pipe), then the process on the left receives the SIGPIPE signal as soon as it tries to write to the pipe. This is done like so to make sure that the pipeline dies quickly if any of the process within it dies.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.