Periodically look at a pipe and extract new output that has been produced since the last time I have looked at it

Question

I am reading a continuously growing output of a pipe (for example output from inotifywait).

I need to periodically examine the pipe output and extract all new output that has been produced since the last time I have looked at it.

How do I do this in a shell script? I'm out ideas of keywords to use for googling.

Answer 1

The data in a pipe can only be read once; the "what's new" part is easy. Just create a named pipe with mkfifo, redirect your inotifywait output to it with > and read the pipe periodically.

The trickier part is reading a pipe, which is open for write somewhere, without blocking. dd can do this.

Here's the setup I used to create and continuously write to a pipe:

mkfifo foo
( while true ; do date; sleep 1 ; done ) > foo

And to read all unread data:

dd iflag=nonblock if=foo bs=1M count=1 of=buffer.txt

You can change of=... to an output file of your choosing.

Sooner or later you will get a partial line from the pipe, so make sure your script can handle this. For the kind of activity you describe, a good approach is to repeat the dd in append mode until the buffer is newline-terminated:

buf=buffer.txt
pipe=foo
> $buf # empty the buffer
until [[ $( tail -c1 $buf | od -a ) == *nl* ]] # nl means newline
do
  dd iflag=nonblock oflag=append conv=notrunc if=$pipe bs=1M count=1 of=$buf
  ls -l $buf # see how it grows
  sleep 1 # if the writer dies, this loop will be infinite and we don't want to kill the CPU
done
do_stuff.sh < $buf
# rm $buf

EDIT: it appears you want to tell inotifywait when you're at the terminal and dump everything that's new. That's easier. Make a file like whatsnew.sh:

#!/bin/bash
echo "waiting for first output ... "
while true
do
    n=0
    while read -t0.1 line
    do
        echo "[$line]"
        (( n++ ))
    done
    read -p "$n new lines.  Press any key to try again... " -n1 -s </dev/tty
    echo
done

Then start it up:

inotifywait | whatsnew.sh