Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

Is there a utility that I can stick in a pipeline to decouple read and write speeds?

$ producer | buf | consumer

Basically, I want a utility buf that reads its input as fast as possible, storing it in memory so consumer can take its sweet time while producer runs as fast as possible.

share|improve this question

4 Answers 4

The pv (pipe viewer) utility can do this (with the -B option) and a lot more, including giving you progress reports.

share|improve this answer
    
Is there a way to do this with an unbounded amount of data? As best as I can tell, I need to supply a number with -B and if the producer gets that far ahead of the consumer, the producer will slow down again. If you're in a situation where there are multiple consumers (producer | tee >(pv -cB $SIZE | consumer1) | pv -cB $SIZE2 | consumer2), this can cause slowdowns again. –  Daniel H Jun 17 '13 at 16:40
    
I've used pv hundreds of times and never knew this. Very awesome, Thank you! –  Rucent88 Aug 8 at 16:40

you can use dd:

producer | dd bs=64K | consumer

It's available on every unix.

share|improve this answer
    
+1 for using standard utility, although pv is probably is probably nicer to use (shows progress). –  Totor Mar 17 '13 at 15:17
    
Does that actually decouple the reading and writing speed? It seems like dd only stores one block at a time, so it would just delay everything by the amount of time it takes to produce the block size; please correct me if I'm wrong. Also, can this buffering be extended to unlimited size, or only whatever's entered for the block size? –  Daniel H Jun 17 '13 at 16:45

Take a look at mbuffer. It can buffer to memory or memory mapped file(-t/-T).

share|improve this answer
    
As I asked for the others, is there a way to tell it to buffer as much as is necessary, or does it have a maximum size? Is there a conceptual reason why most of these programs do have maximum sizes and don't, for example, use a linked list of smaller buffers (or any other arbitrary-size queue implementation)? –  Daniel H Jun 19 '13 at 22:11

This is basically a negative answer. It appears that neither dd, nor mbuffer, nor even pv works is all cases, in particular if the rate of data generated by the producer can vary a lot. I give some testcases below. After typing the command, wait for about 10 seconds, then type > (to go to the end of the data, i.e. wait for the end of the input).

zsh -c 'echo foo0; sleep 3; \
        printf "Line %060d\n" {1..123456}; \
        echo foo1; sleep 5; \
        echo foo2' | dd bs=64K | less

Here, after typing >, one has to wait for 5 seconds, meaning that the producer (zsh script) has blocked before the sleep 5. Increasing the bs size to e.g. 32M doesn't change the behavior, though the 32MB buffer is large enough. I suspect that this is because dd blocks on output instead of going on with the input. Using oflag=nonblock is not a solution because this discards data.

zsh -c 'echo foo0; sleep 3; \
        printf "Line %060d\n" {1..123456}; \
        echo foo1; sleep 5; \
        echo foo2' | mbuffer -q | less

With mbuffer, the problem is that the first line (foo0) doesn't appear immediately. There doesn't seem to be any option to enable line-buffering on input.

zsh -c 'echo foo0; sleep 3; \
        printf "Line %060d\n" {1..123456}; \
        echo foo1; sleep 5; \
        echo foo2' | pv -q -B 32m | less

With pv, the behavior is similar to dd. Worse, I suspect that it does wrong things to the terminal since sometimes less can no longer receive input from the terminal; for instance, one cannot quit it with q.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.