26

I'm looking for something I suspect doesn't exist: A non-blocking buffered named pipe (fifo) for use from the command line. Is there such a thing?

Here's the use case: Suppose I have a process that's going to run a long time in the background and spew a lot of output to stdout. I don't really care about the output and don't want to store it (perhaps I don't have enough room to), but I'd like to "drop in" periodically and follow what it's doing, then drop out again and leave it to do its work. So I'd like to redirect its output to this theoretical buffered, non-blocking named pipe and then periodically tap into it.

So basically I want to start like this (10M being the size of the buffer):

mkmagicfifo magicfifo 10M
spewingprocess > magicfifo &

...and periodically drop in to see what's going on...

tail -f magicfifo

...without magicfifo storing all the output (so, not a normal file), and without it blocking the spewing process when it fills up and isn't tapped (so, not quite a normal named pipe).

I don't think solutions involving tail or prune will do it (well, I can think of a workaround involving tail), because tail would still require that I store all the data somewhere (if I want to drop in and drop out of looking at it), and prune has to rewrite the file, presumably (I'll admit I haven't tried/proven this) breaking the redirection of the process generating all the output.

I expect I could write some utility to do this, but *nix has so many cool aspects of files and pipes and such, I just can't help but think this exists and I just don't know about it.

So: Is there such a thing, and if so what is it?

4
  • 1
    What you are describing is a "ring buffer" or "circular buffer". I'm not aware of any command line tools for maintaining such a thing, although it would be trivial to create. Oct 28, 2011 at 14:50
  • 2
    Have a look at the solutions described in "Linux non-blocking fifo (on demand logging)", stackoverflow.com/questions/7360473/….
    – user11915
    Oct 28, 2011 at 15:45
  • 1
    Looks like this has been solved on StackOverflow: stackoverflow.com/questions/7360473/… Mar 12, 2014 at 17:09
  • @JamesBlackburn: Thanks! Very interesting. Mar 12, 2014 at 17:45

3 Answers 3

16

I think what you are looking for is GNU screen. It maintains a buffer to hold the last screen full or two of output from one or more programs and lets you disconnect and come back later.

4
  • +1 for suggesting screen. BTW, you can configure it to hold lots of "history lines". Oct 28, 2011 at 15:27
  • 1
    Thanks. Can you give an example of how you'd apply that to the commands I showed in my question? The man page says it's a window manager (I think they mean in a terminal sense, not a graphical sense, but still). And would I still be able to drop in (via ssh) and drop out as necessary? (E.g., ops on remote servers.) Oct 28, 2011 at 15:41
  • Yes, you can use GNU screen this way. You would create a new (potentially named) session, run your command inside that session, and then disconnect.
    – TML
    Oct 28, 2011 at 22:46
  • 2
    There's also tmux and dtach -- anything in the same class of terminal multiplexer / session manager application should be able to achieve the same thing.
    – jw013
    Feb 7, 2013 at 18:52
5

You can use pv, it provides as much buffering as you want in a pipeline. You can use it like this:

sprewingprocess | pv -B 1g > ordinaryfifo &

That would give you up to 1GB of buffering between spewingprocess and the fifo. Most Linux distributions offer pv in a package called, believe it or not, pv.

4
  • Thanks, but wouldn't that block once the buffer was full if I wasn't reading the target named pipe? Oct 29, 2011 at 7:28
  • 1
    Yes, but what choice do you have? In a finite universe, you cannot have literally unlimited buffering. Oct 29, 2011 at 21:58
  • The other choice is as I described in my question: Not storing all the output. When the buffer is full, the oldest stuff is thrown away. Oct 30, 2011 at 9:39
  • Hmm, I tested this and it unfortunately does not entirely work. If the process reading the fifo stops reading for a while, pv blocks trying to write to the fifo, and because it's not multithreaded, that also blocks reading data into pv's buffer. So pv's buffer will only continue to fill while the process reading the fifo keeps reading. pv may be able to read and buffer some data, but it doesn't prevent the writer from blocking entirely. Mar 26, 2014 at 22:07
1

I had the same problem. This is my first solution. First write output to a file which we truncate after each line so it doesn't grow indefinitely:

spewingprocess | while read line; do echo $line > buffer.txt ; done

Then read from the file using tail (where 2> /dev/null gets rid of the "file truncated" error message):

tail -f ./buffer.txt 2> /dev/null

This way the buffer doesn't grow and we can multiplex, e.g. run as many tails as we want. However, the problem with this approach is that we can lose data when we truncate faster than tail can read as this test shows:

for ((i=0; ; i++)) ; do echo "$i" ; done | while read line; do  echo $line > buffer.txt ; done
tail -f ./buffer.txt 2> /dev/null > log.txt

After running for some time the first and last lines are:

$ head -n 1 log.txt
0
$ tail -n 1 log.txt
78783

But the file has fewer lines, so some are lost:

$ wc log.txt
67087  67087 392819 log.txt

Still this seems a nice solution if you don't care so much about data loss or when your spewingprocess is not fast enough for data lose to occur.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .