Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I am developing something that consists of a program I made generating output, and a program I did not make (and would very much like not to have to alter it) consuming it. I am using a named FIFO created with mkpipe. I suspect I have a performance bottleneck due to the smallness of the Linux FIFO buffer. (And such size cannot be changed without patching the kernel.)

Is there an alternative FIFO-like thing I can use with a buffer of, say, a couple dozen megabytes? The not-made-by-me program can read from anything that can have a filesystem name - /dev/stdin, a named pipe (like it's doing now), a real file, /dev/any-character-device...

Even a third-party free kernel module could do, if no other alternative is available.

share|improve this question
    
What's wrong with using a file? –  Kyle Jones Jun 4 '12 at 16:46
    
@Kyle: the reading program will think the file is finished and terminate if it catches up with the writing program. –  JCCyC Jun 4 '12 at 18:36
2  
I don't think the problem is in the FIFO. Try this: mkfifo fifo; yes >> fifo & time (head -1000000000 < fifo | wc -c). Then divide the amount of characters (2000000000) by the time (in my case, 23.5 s). I get more than 80 MB/s. What do you get? –  angus Jun 4 '12 at 21:17
    
Please give more information. What makes the consumer decide to terminate? (Try observing it with strace. Post the trace if you have trouble interpreting it.) How much data do you have in total? How long does the processing take? –  Gilles Jun 4 '12 at 23:38
add comment

2 Answers

As shown, there are no problems on named pipes throughput, so if your writer is faster than your reader, and both work at a constant rate, your whole task will be as slow as your reader, no matter what size the buffer is.

But if your writer produces output in bursts, with pauses between them, then a bigger buffer is an advantage.

There is an utility (buffer(1)), meant for writing to tapes, that buffers input and lets it go at a constant rate (by default, as fast as it can).

your-producer | buffer > fifo & your-consumer fifo

It's in Debian package buffer, or just download it from here.

share|improve this answer
add comment

You can use a unix domain socket and configure the buffer size with setsockopt().

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.