I'm looking for something I suspect doesn't exist: A non-blocking buffered named pipe (fifo) for use from the command line. Is there such a thing?

Here's the use case: Suppose I have a process that's going to run a long time in the background and spew a lot of output to stdout. I don't really care about the output and don't want to store it (perhaps I don't have enough room to), but I'd like to "drop in" periodically and follow what it's doing, then drop out again and leave it to do its work. So I'd like to redirect its output to this theoretical buffered, non-blocking named pipe and then periodically tap into it.

So basically I want to start like this (10M being the size of the buffer):

mkmagicfifo magicfifo 10M
spewingprocess > magicfifo &

...and periodically drop in to see what's going on...

tail -f magicfifo

...without magicfifo storing all the output (so, not a normal file), and without it blocking the spewing process when it fills up and isn't tapped (so, not quite a normal named pipe).

I don't think solutions involving tail or prune will do it (well, I can think of a workaround involving tail), because tail would still require that I store all the data somewhere (if I want to drop in and drop out of looking at it), and prune has to rewrite the file, presumably (I'll admit I haven't tried/proven this) breaking the redirection of the process generating all the output.

I expect I could write some utility to do this, but *nix has so many cool aspects of files and pipes and such, I just can't help but think this exists and I just don't know about it.

So: Is there such a thing, and if so what is it?

  • 1
    What you are describing is a "ring buffer" or "circular buffer". I'm not aware of any command line tools for maintaining such a thing, although it would be trivial to create. – Shawn J. Goff Oct 28 '11 at 14:50
  • 2
    Have a look at the solutions described in "Linux non-blocking fifo (on demand logging)", stackoverflow.com/questions/7360473/…. – user11915 Oct 28 '11 at 15:45
  • 1
    Looks like this has been solved on StackOverflow: stackoverflow.com/questions/7360473/… – James Blackburn Mar 12 '14 at 17:09
  • @JamesBlackburn: Thanks! Very interesting. – T.J. Crowder Mar 12 '14 at 17:45

I think what you are looking for is GNU screen. It maintains a buffer to hold the last screen full or two of output from one or more programs and lets you disconnect and come back later.

| improve this answer | |
  • +1 for suggesting screen. BTW, you can configure it to hold lots of "history lines". – Mr Shunz Oct 28 '11 at 15:27
  • 1
    Thanks. Can you give an example of how you'd apply that to the commands I showed in my question? The man page says it's a window manager (I think they mean in a terminal sense, not a graphical sense, but still). And would I still be able to drop in (via ssh) and drop out as necessary? (E.g., ops on remote servers.) – T.J. Crowder Oct 28 '11 at 15:41
  • Yes, you can use GNU screen this way. You would create a new (potentially named) session, run your command inside that session, and then disconnect. – TML Oct 28 '11 at 22:46
  • 2
    There's also tmux and dtach -- anything in the same class of terminal multiplexer / session manager application should be able to achieve the same thing. – jw013 Feb 7 '13 at 18:52

You can use pv, it provides as much buffering as you want in a pipeline. You can use it like this:

sprewingprocess | pv -B 1g > ordinaryfifo &

That would give you up to 1GB of buffering between spewingprocess and the fifo. Most Linux distributions offer pv in a package called, believe it or not, pv.

| improve this answer | |
  • Thanks, but wouldn't that block once the buffer was full if I wasn't reading the target named pipe? – T.J. Crowder Oct 29 '11 at 7:28
  • 1
    Yes, but what choice do you have? In a finite universe, you cannot have literally unlimited buffering. – David Schwartz Oct 29 '11 at 21:58
  • The other choice is as I described in my question: Not storing all the output. When the buffer is full, the oldest stuff is thrown away. – T.J. Crowder Oct 30 '11 at 9:39
  • Hmm, I tested this and it unfortunately does not entirely work. If the process reading the fifo stops reading for a while, pv blocks trying to write to the fifo, and because it's not multithreaded, that also blocks reading data into pv's buffer. So pv's buffer will only continue to fill while the process reading the fifo keeps reading. pv may be able to read and buffer some data, but it doesn't prevent the writer from blocking entirely. – Daniel S. Sterling Mar 26 '14 at 22:07

I had the same problem. This is my first solution. First write output to a file which we truncate after each line so it doesn't grow indefinitely:

spewingprocess | while read line; do echo $line > buffer.txt ; done

Then read from the file using tail (where 2> /dev/null gets rid of the "file truncated" error message):

tail -f ./buffer.txt 2> /dev/null

This way the buffer doesn't grow and we can multiplex, e.g. run as many tails as we want. However, the problem with this approach is that we can lose data when we truncate faster than tail can read as this test shows:

for ((i=0; ; i++)) ; do echo "$i" ; done | while read line; do  echo $line > buffer.txt ; done
tail -f ./buffer.txt 2> /dev/null > log.txt

After running for some time the first and last lines are:

$ head -n 1 log.txt
$ tail -n 1 log.txt

But the file has fewer lines, so some are lost:

$ wc log.txt
67087  67087 392819 log.txt

Still this seems a nice solution if you don't care so much about data loss or when your spewingprocess is not fast enough for data lose to occur.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.