Buffering (named) pipe in GNU OS

Question

In the GNU OS a process can only write data to a pipe if another process reads the same data (from the same pipe) at the same time.

Is there something like a pipe which lets the 1st process write and buffers the data until the 2nd one reads it?

Answer 1

A named pipe(fifo) can do what you want up to a point, but with a couple of caveats:

1. If you want to be able to write to the pipe before the reader exists, your writer must open the fifo as read-write or the call to open will block. In the shell this might look like this:

   exec 3<>/path/to/pipe
   echo "foo" >&3
   cat <&3
   

   As you can see there, I can write to the pipe before the reader is around. Once I do read from the pipe, the messages I've written are already there.

2. Writes to the pipe will eventually block once the pipe fills. According to pipe(7) on linux:

A pipe has a limited capacity. If the pipe is full, then a write(2) will block or fail, depending on whether the O_NONBLOCK flag is set (see below). Different implementations have different limits for the pipe capacity. Applications should not rely on a particular capacity: an application should be designed so that a reading process consumes data as soon as it is available, so that a writing process does not remain blocked.

In Linux versions before 2.6.11, the capacity of a pipe was the same as the system page size (e.g., 4096 bytes on i386). Since Linux 2.6.11, the pipe capacity is 65536 bytes.

Depending on your use case, you might also consider using a message queue. The kernel provides a message queue. On Linux, see mq_overview(7) for a good overview. Alternatively, services like RabbitMQ provide message queues with a variety of feature sets including being usable across a network.

Answer 2

This question is rather old now - but the buffer command provides the ability to, well, buffer data in a pipe.

Answer 3

Do you mean http://en.wikipedia.org/wiki/Named_pipe ? It works as you asking for.