Bash tail -f with while-read and pipe hangs

Question

In Bash, piping tail -f to a read loop blocks indefinitely.

while read LINE0 
do 
    echo "${LINE0}"; 
done < <( tail -n 3 -f /tmp/file0.txt | grep '.*' ) 
# hangs

Remove the -f or | grep '.*', then the loop will iterate.

The following does not hang.

tail -n 3 -f /tmp/file0.txt | grep '.*' 

What causes this behavior?

Is there anyway in Bash to follow a file and read in a pipe expression?

Answer 1

In the pipeline, grep's output is buffered. With the GNU implementation of grep, you can force flush the output after each line with --line-buffered (documentation here); for example:

tail -n 3 -f /tmp/file0.txt | grep --line-buffered '.*' |
  while IFS= read -r LINE0 
  do 
    printf '%s\n' "${LINE0}"
  done  

Answer 2

Greg's Wiki has a comprehensive article about buffering: https://mywiki.wooledge.org/BashFAQ/009

The technique that worked for me (Popos 20.04, Ubuntu 20.04) was using stdbuf to do line buffering. Their example:

tail -f logfile | stdbuf -oL grep 'foo bar' | awk ...

My use case:

journalctl --output=json -t poke-stats -f |\
 stdbuf -oL jq -r '.__REALTIME_TIMESTAMP' |\
 stdbuf -oL awk '{print $1-last;last=$1}'