Possible Duplicate:
Bash while loop and reading from pipe

I come from web programming background, and find myself interested in one peculiarity of using a local shell. I understand that when a program reads from file, it can read at whatever rate's necessary. But I'm curious to know, how does it work when a program gets other program's input piped in and can't process it in realtime?

A good example would be video encoding. Suppose I point a decoder to video file, then it's output gets piped in as encoder's input. The total size of decoded video is more than ram+swap, so I guess there's no way to have it buffered altogether. I found about read and write calls to stdin and stdout, but I'm interested to know what actually happens when this example's encoder can't cope with all the data at once. Does it somehow informs the decoder about desired rate? Does decoder program needs to be specifically prepared for such a signal and modify it's processing speed accordingly? If not, how does it all gets balanced in the end?

  • +1 for thinking about how things work. Always encouraging to see.
    – itsbruce
    Oct 29, 2012 at 14:36

2 Answers 2


When a writer writes to a pipe and the pipe is full (its size is limited to a few kilobytes), its process blocks until one of the readers frees some space. Similarly, when a reader reads from a pipe, its process blocks until there is something there.

There are also asynchronous writes and reads that a programmer can use to queue up these reads and writes.

I highly suggest reading Beej's Gudes, starting with Beej's Guide to Unix Interprocess Communication

You call it with two arguments, the first is a sleep time for the writer, the second is a a sleep time for the reader. Try calling it with args 0 3 and 3 0

  echo writing output >&2
  echo hi
  sleep $write_sleep
  echo writing output >&2
  echo hi
  sleep $write_sleep
  echo writing output >&2
  echo hi
  sleep $write_sleep
  while true; do
    echo getting input >&2
    read input
    [ $input ] || { echo input is empty >&2 &&  break; }
    echo $input
    sleep $read_sleep

writer | reader

Pipes are usual unix descriptors at the programmatic level. When you setup two programs to communicate through a pipe, all they see at first is the standard output and standard input descriptors, which they interact with, because the shell did setup them that way. There are some peculiarities with these descriptors when setup this way, but processes interact with them as with any other file descriptor (it's the whole unix philosophy, you know).

Basically, the writer process may write as much as he wants in the pipe, but, because the pipe has a maximum storage limit (a buffer), it may be blocked if the pipe is full, or the request may be answered as "try again later" if the writer didn't want it to block.

Conversely, the reader may read as much as he wants, but it may also be blocked (or answered with "try again later" as well) if the pipe is empty.

Not the answer you're looking for? Browse other questions tagged or ask your own question.