Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

When I just used pipe in bash, I didn't think more about this. But when I read some C code example using system call pipe() together with fork(), I wonder how to understand pipes, including both anonymous pipes and named pipes.

It is often heard that "everything in Linux/Unix is a file". I wonder if a pipe is actually a file so that one part it connects writes to the pipe file, and the other part reads from the pipe file? If yes, where is the pipe file for an anonymous pipe created? In /tmp, /dev, or ...?

However, from examples of named pipes, I also learned that using pipes has space and time performance advantage over explicitly using temporary files, probably because there are no files involved in implementation of pipes. Also pipes seem not store data as files do. So I doubt a pipe is actually a file.

share|improve this question
add comment

4 Answers 4

up vote 17 down vote accepted

About your performance question, pipes are more efficient than files because no disk IO is needed. So cmd1 | cmd2 is more efficient than cmd1 > tmpfile; cmd2 < tmpfile (this might not be true if tmpfile is backed on a RAM disk or other memory device as named pipe; but if it is a named pipe, cmd1 should be run in the background as its output can block if the pipe becomes full). If you need the result of cmd1 and still need to send its output to cmd2, you should cmd1 | tee tmpfile | cmd2 which will allow cmd1 and cmd2 to run in parallel avoiding disk read operations from cmd2.

Named pipes are useful if many processes read/write to the same pipe. They can also be useful when a program is not designed to use stdin/stdout for its IO needing to use files. I put files in italic because named pipes are not exactly files in a storage point of view as they reside in memory and have a fixed buffer size, even if they have a filesystem entry (for reference purpose). Other things in UNIX have filesystem entries without being files: just think of /dev/null or others entries in /dev or /proc.

As pipes (named and unnamed) have a fixed buffer size, read/write operations to them can block, causing the reading/writing process to go in IOWait state. Also, when do you receive an EOF when reading from a memory buffer ? Rules on this behavior are well defined and can be found in the man.

One thing you cannot do with pipes (named and unnamed) is seek back in the data. As they are implemented using a memory buffer, this is understandable.

About "everything in Linux/Unix is a file", I do not agree. Named pipes have filesystem entries, but are not exactly file. Unnamed pipes do not have filesystem entries (except maybe in /proc). However, most IO operations on UNIX are done using read/write function that need a file descriptor, including unnamed pipe (and socket). I do not think that we can say that "everything in Linux/Unix is a file", but we can surely say that "most IO in Linux/Unix is done using a file descriptor".

share|improve this answer
    
Thanks! Are the two commands connected by a pipe running in parallel, instead of the second starts to run after the first finishes? –  Tim Aug 5 '11 at 17:15
    
Yes, the 2 commands are run in parallel. If they were not and the 1st output more than the buffer, it would be blocked. You can try it by running cmd1 > fifo and cmd2 < fifo in 2 different shells, creating the named pipe with mkfifo fifo. –  jfgagne Aug 5 '11 at 17:31
    
Another test you can do, is to kill cmd2 while cmd1is still running: cmd1 will probably stop reporting a broken pipe mesage. –  jfgagne Aug 5 '11 at 17:33
    
Thanks! what do you mean would be blocked? If this happens, does it mean the date in the stream after block will be lost? –  Tim Aug 5 '11 at 17:51
2  
Data is not lost. If the pipe buffer is full, cmd1's write to the pipe will only return when cmd2 will have read data from the pipe. In the same way, cmd2's read from a pipe will block if the buffer is empty until cmd1 writes to the pipe. –  jfgagne Aug 5 '11 at 18:00
show 2 more comments

Two of the basic fundamentals of UNIX philosophy are

  1. To make small programs that do one thing well.
  2. and expect the output of every program to become the input to another,as
    yet unknown,program.

    The use of pipes let you leverage the effects of these two design
    fundamentals to create extremely powerful chains of commands to achieve your desired result.

    Most command-line programs that operate on files can also accept input on standard in(input through keyboard) and output to standard out(print on
    screen).

    Some commands are designed to only operate within a pipe can't operate on files directly.

    for example tr command

  ls -C | tr 'a-z' 'A-Z'
    cmd1 | cmd2
  • Sends STDOUT of cmd1 to STDIN of cmd2 instead of the screen.

  • STDERR is not forwarded across pipes.

    In short Pipes is character (|) can connect commands.

    Any command that writes to STDOUT can be be used on the left hand side of pipe.

       ls - /etc | less 
    

    Any command that reads from STDIN can be used on the right-hand side of a pipe.

       echo "test print" | lpr 
    

    A traditional pipe is "unnamed" because it exists anonymously and persists only for as long as the process is running. A named pipe is system-persistent and exists beyond the life of the process and must be deleted once it is no longer being used. Processes generally attach to the named pipe (usually appearing as a file) to perform inter-process communication (IPC).

source : http://en.wikipedia.org/wiki/Named_pipe

share|improve this answer
add comment

To supplement the other answers...

stdin and stdout are file descriptors and are read and written as if they are files. therefor you can do echo hi | grep hi, and it will replace echo's stdout with a pipe and replace stdin of grep to other end of this pipe.

share|improve this answer
add comment

stdin and stdout are file descriptors and are read and written as if they are files. therefor you can do echo hi | grep hi, and it will replace echo's stdout with a pipe and replace stdin of grep to other end of this pipe.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.