mkfifo - Does disk I/O actually occur?

Question

I have 2 applications:

• Producer (N instances)
• Consumer (1 instance)

I currently write out intermediate results from the producers, and then the consumer reads these files from disk and produces a final result.

I would like to minimize this I/O by "streaming" the outputs from the producers directly to the consumer.

I came across named pipes (mkfifo) and a subsequent example here. This looks great, but what I can't determine is how this is actually implemented? Is the FIFO queue just being buffered through a file? If so, that probably wouldn't help me. I would like for the contents to stream "through memory" entirely without utilizing the disk. Maybe this isn't possible across processes?

Answer 1

No disk i/o (except perhaps when navigating through the filesystem to open the fifo file.)

From the Linux fifo(7) man page:

A FIFO special file (a named pipe) is similar to a pipe, except that it is accessed as part of the filesystem. [...] When processes are exchanging data via the FIFO, the kernel passes all data internally without writing it to the filesystem. Thus, the FIFO special file has no contents on the filesystem; the filesystem entry merely serves as a reference point so that processes can access the pipe using a name in the filesystem.

Answer 2

It doesn't really matter whether your result is actually disk backed or not, because if enough memory is available it will anyway be cached and no actual disk IO is performed. On the contrary, if it is memory-backed and not enough memory is available it could be swapped to disk.

If I were to guess, I would say that pipe is actually memory-based, but then, this should only change whether queued data is preserved between reboots.

What you should take care of is that since you have multiple producers your writes need to be atomic so that they are not interleaved in the queue. See man 7 pipe for details about how to ensure that a write is atomic.