Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

Is there a utility that I can stick in a pipeline to decouple read and write speeds?

$ producer | buf | consumer

Basically, I want a utility buf that reads its input as fast as possible, storing it in memory so consumer can take its sweet time while producer runs as fast as possible.

share|improve this question
add comment

3 Answers

The pv (pipe viewer) utility can do this (with the -B option) and a lot more, including giving you progress reports.

share|improve this answer
    
Is there a way to do this with an unbounded amount of data? As best as I can tell, I need to supply a number with -B and if the producer gets that far ahead of the consumer, the producer will slow down again. If you're in a situation where there are multiple consumers (producer | tee >(pv -cB $SIZE | consumer1) | pv -cB $SIZE2 | consumer2), this can cause slowdowns again. –  Daniel H Jun 17 '13 at 16:40
add comment

you can use dd:

producer | dd bs=64K | consumer

It's available on every unix.

share|improve this answer
    
+1 for using standard utility, although pv is probably is probably nicer to use (shows progress). –  Totor Mar 17 '13 at 15:17
    
Does that actually decouple the reading and writing speed? It seems like dd only stores one block at a time, so it would just delay everything by the amount of time it takes to produce the block size; please correct me if I'm wrong. Also, can this buffering be extended to unlimited size, or only whatever's entered for the block size? –  Daniel H Jun 17 '13 at 16:45
add comment

Take a look at mbuffer. It can buffer to memory or memory mapped file(-t/-T).

share|improve this answer
    
As I asked for the others, is there a way to tell it to buffer as much as is necessary, or does it have a maximum size? Is there a conceptual reason why most of these programs do have maximum sizes and don't, for example, use a linked list of smaller buffers (or any other arbitrary-size queue implementation)? –  Daniel H Jun 19 '13 at 22:11
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.