18

Let's say I have a makefile with a recipe called hour_long_recipe, which as its name suggests takes an hour to run. At random points throughout the recipe it asks yes/no questions. Let's say it asks 10 questions in total.

One possible (and often recommended) way to run it is:

yes | make hour_long_recipe

which answers all questions with y. However, from my understanding, yes keeps outputting to its stdout at up to 10.2 GiB per second regardless of whether make is actually using that data from its stdin.

Even if it was just 10 MiB/s (much slower than any implementation of yes if that reddit thread is to be believed), during the course of an hour it would add up to over 35 GiB, of which just 20 bytes will be read. Where does the data go? It's possible to save it to disk but that's wasteful, and if the disk fills up fast enough it can even cause make to fail.

Presumably the operating system will stop it from getting to that, but how? What is the limit, and what happens when that limit is reached?

11
  • 4
    That's a good idea if this was actually something I needed to do, but this question is about understanding how Linux piping works and whether yes will crash or spam your disk in such a situation
    – NeatNit
    Dec 31, 2020 at 13:52
  • 1
    @muru it does, but it took me until fully writing this question (and then some) to figure out that that's the question I probably should have been asking.
    – NeatNit
    Dec 31, 2020 at 15:16
  • 4
    to clarify, this question isn't "how big is the pipe", but rather "what happens when trying to write boatloads of data to a pipe". The fact that a pipe has a maximum size at all is the answer, together with an explanation of what happens when that size is full. Chris's answer is perfect, and includes info not found in the other question you linked.
    – NeatNit
    Dec 31, 2020 at 15:32
  • That's still what you're asking at the end of the question, though.
    – muru
    Dec 31, 2020 at 23:21
  • 1
    @G-ManSays'ReinstateMonica' I don't see how my question is remotely a "duplicate" of that question. The question you linked is very convoluted and stems from a whole bunch of misconceptions and confusion, and the answer is appropriately also complex and covers many topics related to pipes. My question is not addressed, at least not obviously or directly. My question is (I think) concise and specific, with a single correct answer. How is that a duplicate?
    – NeatNit
    Jan 4, 2021 at 13:38

1 Answer 1

35

tl;dr: at some point, yes will be blocked from writing if the data isn't being read on the other side. It will not be able to continue executing until either that data is read, or it receives a signal, so you typically don't need to worry about yes writing gigabytes and gigabytes of data.


The important thing to remember is that a pipe is a FIFO data structure, not simply a pure stream which drops data if not immediately read on the receiver. That is, while it may appear in most cases to be a seamless stream of data from the writing application to the reading application, it does require intermediate storage to perform that, and that intermediate storage is of finite size.*

If we look at the pipe(7) man page, we can read the following about the size of that internal buffer (emphasis added):

In Linux versions before 2.6.11, the capacity of a pipe was the same as the system page size (e.g., 4096 bytes on i386). Since Linux 2.6.11, the pipe capacity is 16 pages (i.e., 65,536 bytes in a system with a page size of 4096 bytes). Since Linux 2.6.35, the default pipe capacity is 16 pages, but the capacity can be queried and set using the fcntl(2) F_GETPIPE_SZ and F_SETPIPE_SZ operations.

Assuming you're using a standard x86_64 system, it's very likely that you use 4KiB pages, so the 2^16 upper limit on pipe capacity likely is correct unless either side of the pipeline at some point used fcntl(F_SETPIPE_SZ). Either way, the principle stands: the intermediate storage between two sides of a pipe is finite, and is stored in memory.

In an abstract pipeline a | b, this storage is used in the period between a writing some data, and b actually reading it. Assuming, then, that your make invocation (and any children also connected to this pipe by inheritance) don't actually try to read stdin, or only do so sparingly, the write syscall from yes will eventually simply not wake up yes from sleep when buffer space is exhausted. yes will then wait to be woken up, either when buffer space is available again, or a signal is received.** All of this is handled by the kernel's process scheduler. You can see this in pipe_write(), which is the write() handler for pipes:

static ssize_t
pipe_write(struct kiocb *iocb, struct iov_iter *from)
{
    /* ... */

    if (pipe_full(pipe->head, pipe->tail, pipe->max_usage))
        wake_next_writer = false;

    if (wake_next_writer)
        wake_up_interruptible_sync_poll(&pipe->wr_wait, EPOLLOUT | EPOLLWRNORM);

    /* ... */
}

When the make side eventually terminates, yes will be sent SIGPIPE as a result of writing to a pipe with nothing remaining on the other end. This will then — depending on yes implementation — invoke either its own signal handler or the default kernel signal handler, and it will terminate.***


* In simple circumstances, where the receiver is processing the data at roughly the same rate that it's being written, this transfer can also be zero-copy with no intermediate buffer by using virtual memory to map and make available the physical page from the writing process available to the receiver. However, the case you're describing will certainly eventually need to use the pipe buffer to store the unread data.

** It's also possible that the writing is done with the O_NONBLOCK flag set on the file descriptor, which enables non-blocking mode. In this case, you'll probably get one incomplete write, and then write will return EAGAIN and the application will need to deal with that itself. It will likely either do that by suspending or running some other code of its choosing to handle the pipe being full. In the case of every modern yes version I can find and most other applications, though, the description above is what happens, since they don't use O_NONBLOCK.

*** An application can do whatever it likes upon receiving SIGPIPE -- it may even theoretically decide not to terminate. However, all common yes use the default SIGPIPE handler, which just terminates without executing any more userspace instructions.

9
  • 2
    That's in essence the same as a real life pipe. Dec 31, 2020 at 19:36
  • 8
    The technical term for this is "blocking". write blocks when the pipe's buffer is full, and doesn't return until the reader has reduced the buffer contents sufficiently for write to store all the data it was given in the buffer.
    – cjm
    Jan 1, 2021 at 3:44
  • 1
    Another related technical term is "flow control". And yes, blocking the writer is part of the implementation of how flow control works on a pipe. (Though in general, if the writer is something other than yes, it might also be possible that the writer sets its end of the pipe to O_NONBLOCK mode, and then integrates it into a select/poll/epoll/... loop.) Jan 1, 2021 at 21:03
  • Do you know of any yes implementation which installs its own SIGPIPE signal handler? If not, that statement is both superfluous and incomplete, because a hyphothetical yes implementation could also ignore the SIGPIPE signal and handle (or just error out on) the errno = EPIPE from write(2) (as python does).
    – user313992
    Jan 11, 2021 at 15:38
  • @user414777 yes isn't standardised, so sure, you can write any kind of behaviour you want. I don't really have much interest in getting into semantic hypotheticals, but it's not farfetched to believe that there may be some yes which does some kind of teardown prior to terminating. It is, of course, possible to create a yes which doesn't operate at all like a "normal yes" (ie. GNU, BSD, popular historic Unices), but at that point this discussion ceases to have any real meaning. But sure, I can add a clarification that this is not enforced by the kernel.
    – Chris Down
    Jan 11, 2021 at 17:35

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .