Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I've never really thought about how the shell actually executes piped commands. I've always been told that the "stdout of one program gets piped into the stdin of another," as a way of thinking about pipes. So naturally, I thought that in the case of say, A | B, A would run first, then B gets the stdout of A, and uses the stdout of A as its input.

But I've noticed that when people search for a particular process in ps, they'd include grep -v "grep" at the end of the command to make sure that grep doesn't appear in the final output. This means that in the command ps aux | grep "bash" | grep -v "grep", which means that ps knew that grep was running and therefore is in the output of ps. But if ps finishes running before its output gets piped to grep, how did it know that grep was running?

flamingtoast@FTOAST-UBUNTU: ~$ ps | grep ".*"
PID TTY          TIME CMD
3773 pts/0    00:00:00 bash
3784 pts/0    00:00:00 ps
3785 pts/0    00:00:00 grep
share|improve this question
2  
The pipeline's pieces run concurrently. It's not like one piece runs then the next. –  jw013 Apr 28 '12 at 3:37
add comment

4 Answers 4

The order the commands are run actually doesn't matter and isn't guaranteed. Leaving aside the arcane details of pipe(), fork(), dup() and execve(), the shell first creates the pipe, the conduit for the data that will flow between the processes, and then creates the processes with the ends of the pipe connected to them. The first process that is run may block waiting for input from the second process, or block waiting for the second process to start reading data from the pipe. These waits can be arbitrarily long and don't matter. Whichever order the processes are run, the data eventually gets transferred and everything works.

share|improve this answer
2  
Nice answer, but the OP seems to think the processes run sequentially. You might make it clearer here that the processes are run concurrently, and the pipe is like.... a pipe between buckets, where water flows through all at the (approx.) same time. –  Keith Apr 28 '12 at 6:16
    
Thank you for the clarification. The sources I've been reading made it seem like piped programs ran sequentially, rather than concurrently. –  action_potato Apr 28 '12 at 7:58
add comment

Piped commands run concurrently. When you run ps | grep …, it's the luck of the draw (or a matter of details of the workings of the shell combined with scheduler fine-tuning deep in the bowels of the kernel) as to whether ps or grep starts first, and in any case they continue to execute concurrently.

This is very commonly used to allow the second program to process data as it comes out from the first program, before the first program has completed its operation. For example

grep pattern very-large-file | tr a-z A-Z

begins to display the matching lines in uppercase even before grep has finished traversing the large file.

grep pattern very-large-file | head -n 1

displays the first matching line, and may stop processing well before grep has finished reading its input file.

If you read somewhere that piped programs run in sequence, flee this document. Piped programs run concurrently and always have.

share|improve this answer
2  
And what's cool about this example is that when head gets the one line it needs, it terminates and when grep notices this, it also terminates without doing a bunch of further work for nothing. –  Joe May 5 '12 at 20:48
    
I guess there is some kind of IO buffer concerning the pipe... how do I know it's size in bytes? What do I want to read to learn more about it? :) –  naxa Nov 24 '13 at 21:14
1  
@naxa There are two buffers, actually. There's the stdio buffer inside the grep program, and there's a buffer managed by the kernel in the pipe itself. For the latter, see How big is the pipe buffer? –  Gilles Nov 24 '13 at 21:29
add comment

Typically you run this under bash. process working and starting concurrently, but are running by the shell in parallel. How is it possible?

  1. if it isn't last command in pipe, create unnamed pipe with pair of sockets
  2. fork
  3. in child reassign stdin/stdout to sockets if it's needed (for first process in pipe stdin is not reassigned, the same for last process and his stdout)
  4. in child EXEC specified command with arguments that sweep out original shell code, but leaves all opened by them sockets. child process ID will not be changed because this is the same child process
  5. concurrently with child but parallel under main shell go to step 1.

system not guarantee how fast exec will be executed and specified command starts. it's independent to the shell, but system. This is because:

ps auxww| grep ps | cat

once show grep and/or ps command, and next now. It depends how fast kernel really start processes using system exec function.

share|improve this answer
    
Concurrent execution means that two or more processes execute within the same time frame, usually with some sort of dependency between them. Parallel execution means that two or more processes execute simultaneously (e.g. on separate CPU cores at the same time). Parallelism is not relevant to the question, nor is "how fast" exec() is executed, but how the exec() calls and execution of the programs in a pipe are interleaved. –  Thomas Nyman Sep 6 '13 at 10:25
add comment

At the risk of beating a dead horse, the misconception seems to be that

    A | B

is equivalent to

    A > temporary_file
    B < temporary_file
    rm temporary_file

But, back when Unix was created and children rode dinosaurs to school, disks were very small, and it was common for a rather benign command to consume all the free space in a file system.  If B was something like grep some_very_obscure_string, the final output of the pipeline could be much smaller than that intermediate file.  Therefore, the pipe was developed, not as a shorthand for the “run A first, and then run B with input from A’s output” model, but as a way for B to execute concurrently with A and eliminate the need for storing the intermediate file on disk.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.