16

If I have a pipe:

process1 | process2

And process 1 generates gigabytes and gigabytes of data very quickly, but process2 needs to send that data across a network, and is therefore much slower, does this:

  • Slow down process1's execution?; OR
  • Buffer the data somewhere, until process 2 can read it?

If the data is being buffered, is it by the kernel? Is it in memory, or on disk? How big is this buffer? What happens when the buffer overflows?

5
  • 2
    Do man pipe and man 7 pipe,
    – NickD
    Commented Oct 14, 2022 at 1:49
  • 3
  • 2
    See How do pipelines limit memory usage? Commented Oct 14, 2022 at 4:30
  • 1
    Does this slow down process1, or buffer the data somewhere? Both. The buffer is of finite size, so once it fills up, process1 blocks on writing until enough space is available. (There's almost no difference: process2 blocks while writing the slow network, and process1 blocks while writing the pipe to the slow/blocked process2.) Commented Oct 15, 2022 at 16:38
  • If I'm not making a logical error it doesn't matter much now, does it? The wall clock time needed to process the data will not vary much, whether the data is quickly written in a large buffer by process1 and then slowly processed by process2, or whether process1 must wait. The slower process determines the run time. (Now no buffering at all could be inefficient if the speed of process1 varies and sometimes is faster than process2, so that the normally slower process suddenly must wait). Likewise, the CPU usage will be the same (a waiting process does not consume CPU time). Commented Oct 16, 2022 at 17:33

1 Answer 1

25

The slower process limits the speed of the faster process. A pipe is a buffer (512 to 64k bytes in size, depending on the kernel and any adjustments by the process using fcntl(2) operations) between a process that's putting bytes into the buffer with write() calls, and another process that's pulling bytes out of the buffer with read() calls.

Both the write() and read() calls pass control to the kernel. So if the reading process calls read() on an empty pipe buffer, the kernel won't return from the read() until the other process puts bytes in (or closes its stdout file descriptor). At the other end, if the writing process calls write() on a full pipe buffer, the kernel won't add the new bytes to the buffer and return from the write() until the other process pulls bytes out (or closes its stdin file descriptor).

So the effect is that the faster process's performance is constrained by the performance of the slower process. The kernel doesn't allow the pipe buffer to overflow, nor to underflow.

5
  • Note that the pipe buffer size varies wildly from kernel to kernel. "Around 1k" is right, but the real range is 512b - 64k, looking at various versions of BSD and Linux. I recall some older unixes use the page size for pipe buffer size, but this actually seems uncommon today.
    – user10489
    Commented Oct 14, 2022 at 4:12
  • 2
    On modern-ish Linux, the default size is 64KiB on most hardware but can be modified, see man 7 pipe: Since Linux 2.6.11, the pipe capacity is 16 pages (i.e., 65,536 bytes in a system with a page size of 4096 bytes). Since Linux 2.6.35, the default pipe capacity is 16 pages, but the capacity can be queried and set using the fcntl(2) F_GETPIPE_SZ and F_SETPIPE_SZ operations.
    – marcelm
    Commented Oct 14, 2022 at 9:40
  • marcelm and user10489 thank you, I'll edit my answer
    – Sotto Voce
    Commented Oct 14, 2022 at 11:26
  • 1
    If this causes you a problem, there is a tool called pv that will let you set an in-memory buffer of any size to allow the first process to run at full speed. Commented Oct 16, 2022 at 7:27
  • 1
    Apparently you can change the buffer size of a pipe with an fcntl call up to the limit specified in /proc/sys/fs/pipe−max−size which, in turn, can be change with sufficient privileges; these days it appears to be 1 MB by default in Linux. Commented Oct 16, 2022 at 17:25

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .