0

I am reading a continuously growing output of a pipe (for example output from inotifywait).

I need to periodically examine the pipe output and extract all new output that has been produced since the last time I have looked at it.

How do I do this in a shell script? I'm out ideas of keywords to use for googling.

  • A FIFO buffer may be better to use than a pipe - what exactly are you trying to do? – ivanivan Mar 7 '18 at 11:56
  • I am looking at inotifywait output, exactly like in the example. I mentioned pipe as I imagine next I should pipe that output to something to process it. The output comes in bursts: few lines now, another chunk of lines several hours later and so on. Instead staring at it in the terminal, I want to have the output periodically examined to determine what's new since last examination and do something with these new lines (save to a timestamp-pattern file, email it etc), the key question being how do I identify these new lines using a shell script or any other linux utilities – Costin Gușă Mar 7 '18 at 12:46
  • Could this post help you? You can try to have your output be concatenated to a file and then have some function read the file use the output as new input for another command. – kemotep Mar 9 '18 at 14:14
3
+50

The data in a pipe can only be read once; the "what's new" part is easy. Just create a named pipe with mkfifo, redirect your inotifywait output to it with > and read the pipe periodically.

The trickier part is reading a pipe, which is open for write somewhere, without blocking. dd can do this.

Here's the setup I used to create and continuously write to a pipe:

mkfifo foo
( while true ; do date; sleep 1 ; done ) > foo

And to read all unread data:

dd iflag=nonblock if=foo bs=1M count=1 of=buffer.txt

You can change of=... to an output file of your choosing.

Sooner or later you will get a partial line from the pipe, so make sure your script can handle this. For the kind of activity you describe, a good approach is to repeat the dd in append mode until the buffer is newline-terminated:

buf=buffer.txt
pipe=foo
> $buf # empty the buffer
until [[ $( tail -c1 $buf | od -a ) == *nl* ]] # nl means newline
do
  dd iflag=nonblock oflag=append conv=notrunc if=$pipe bs=1M count=1 of=$buf
  ls -l $buf # see how it grows
  sleep 1 # if the writer dies, this loop will be infinite and we don't want to kill the CPU
done
do_stuff.sh < $buf
# rm $buf

EDIT: it appears you want to tell inotifywait when you're at the terminal and dump everything that's new. That's easier. Make a file like whatsnew.sh:

#!/bin/bash
echo "waiting for first output ... "
while true
do
    n=0
    while read -t0.1 line
    do
        echo "[$line]"
        (( n++ ))
    done
    read -p "$n new lines.  Press any key to try again... " -n1 -s </dev/tty
    echo
done

Then start it up:

inotifywait | whatsnew.sh
  • the "what's new" part is easy <- do you think you can explain that as well? I believe inotifywait will print only LF terminated lines; my primary interest of the question is in the "what's new" detection logic / script. I might have misstated the question, thinking I can only do this with reading the output as a pipe (inotifywait | whatsnew.sh). reading from a file to which I'd be redirecting the inotifywait output would do that as well (whatsnew.sh /path/to/output) – Costin Gușă Mar 12 '18 at 6:56
  • It's easy because reading the output consumes it. What has been read cannot be read again. Whatever you do read will be new. My answer describes reading a named pipe, not a regular file. Reading from a pipeline is the same except the piped-to process never exits. It doesn't matter that it prints only complete lines; the printing takes non-zero time and sooner or later your read and inotifywait's write will occur simultaneously. If you perform blocking waits (i.e. normal reads), the I/O will be buffered and you will not get partial lines. See my edits. – Oh My Goodness Mar 12 '18 at 10:40
  • Thank you, we are on the right track, but again I must have been misunderstood for the following: insted of effectively looking at the output I really meant the script to "look" at the output, not the human to look, so I can further send that processed output over email. Apologies, english is not my native language. So instead of manually responding to read call, I am looking to periodically (let's say once a minute) have the script examine the pipe. Also, I am observing you are reading with a timeout of 0.1 seconds. What if the pipe is delivering data at a much bigger rate, can lines be lost? – Costin Gușă Mar 12 '18 at 13:07
  • If you want to check every minute, replace the read -p ... prompt with sleep 60. The while .. done loop is one check, and it can be piped to a program like mail, e.g. while ... done | Mail -s output me@example.com or redirected to a file while ... done > output.txt. The capacity of a pipe is probably 64KB on your system. If it overflows, the writer will block (wait) on write. If you're worried about that, send inotifywait output to a file, and use tail -f inotifywait_output.txt | whatsnew.sh to do the periodic checking. – Oh My Goodness Mar 12 '18 at 20:08
  • I replaced the read line with sleep 60, but now I am getting a blank new line as output from whatsnew.sh. I did not expect that, how do I get a new line only with effective new output from the pipe, and not blank new lines for each sleep? I definitely do not want to get a blank notification for each cycle check. Also still waiting for the answer to the question related to the 0.1 timeout and potential lost lines in between two cycle runs. Thank you! – Costin Gușă Mar 13 '18 at 16:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.