6

In Bash, piping tail -f to a read loop blocks indefinitely.

while read LINE0 
do 
    echo "${LINE0}"; 
done < <( tail -n 3 -f /tmp/file0.txt | grep '.*' ) 
# hangs

Remove the -f or | grep '.*', then the loop will iterate.

The following does not hang.

tail -n 3 -f /tmp/file0.txt | grep '.*' 

What causes this behavior?

Is there anyway in Bash to follow a file and read in a pipe expression?

4
  • If you use -f, it is supposed to "hang" in the sense that tail keeps waiting for more lines to be added to the file. When I try tail -n 3 -f /tmp/file0.txt | wc, it hangs for me. It should hang because wc never gets an end-of-file signal. Are you sure that it doesn't hang for you?
    – John1024
    Commented Sep 12, 2016 at 21:14
  • That's a good point. I corrected my question, replacing wc with grep. Commented Sep 12, 2016 at 21:18
  • grep may not "hang" but it does have buffering issues. It may not print anything until there is enough to fill a buffer.
    – John1024
    Commented Sep 12, 2016 at 21:21
  • Thanks @John1024! I demonstrated that you are correct by: I added lines to the file, and the loop finally iterated. It looks like the buffering is not in the tail -f, but in the read. If I grep for an expression that is only in a couple of lines, even if the file is grown large, the loop will not iterate. I now need to figure out how to stop read from buffering. Commented Sep 12, 2016 at 21:34

2 Answers 2

18

In the pipeline, grep's output is buffered. With the GNU implementation of grep, you can force flush the output after each line with --line-buffered (documentation here); for example:

tail -n 3 -f /tmp/file0.txt | grep --line-buffered '.*' |
  while IFS= read -r LINE0 
  do 
    printf '%s\n' "${LINE0}"
  done  
0
1

Greg's Wiki has a comprehensive article about buffering: https://mywiki.wooledge.org/BashFAQ/009

The technique that worked for me (Popos 20.04, Ubuntu 20.04) was using stdbuf to do line buffering. Their example:

tail -f logfile | stdbuf -oL grep 'foo bar' | awk ...

My use case:

journalctl --output=json -t poke-stats -f |\
 stdbuf -oL jq -r '.__REALTIME_TIMESTAMP' |\
 stdbuf -oL awk '{print $1-last;last=$1}'

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .