Alternate FIFO device for Linux with a (way) bigger buffer while still having filesystem semantics?

Question

I am developing something that consists of a program I made generating output, and a program I did not make (and would very much like not to have to alter it) consuming it. I am using a named FIFO created with mkpipe. I suspect I have a performance bottleneck due to the smallness of the Linux FIFO buffer. (And such size cannot be changed without patching the kernel.)

Is there an alternative FIFO-like thing I can use with a buffer of, say, a couple dozen megabytes? The not-made-by-me program can read from anything that can have a filesystem name - /dev/stdin, a named pipe (like it's doing now), a real file, /dev/any-character-device...

Even a third-party free kernel module could do, if no other alternative is available.

Answer 1

As shown, there are no problems on named pipes throughput, so if your writer is faster than your reader, and both work at a constant rate, your whole task will be as slow as your reader, no matter what size the buffer is.

But if your writer produces output in bursts, with pauses between them, then a bigger buffer is an advantage.

There is an utility (buffer(1)), meant for writing to tapes, that buffers input and lets it go at a constant rate (by default, as fast as it can).

your-producer | buffer > fifo & your-consumer fifo

It's in Debian package buffer, or just download it from here.

Answer 2

You can use a unix domain socket and configure the buffer size with setsockopt().