Piping from grep to awk not working

Question

I am trying to grep the ongoing tail of file log and get the nth word from a line. Example file:

$ cat > test.txt <<EOL
Beam goes blah
John goes hey
Beam goes what?
John goes forget it
Beam goes okay
Beam goes bye
EOL
^C

Now if I do a tail:

$ tail -f test.txt
Beam goes blah
John goes hey
Beam goes what?
John goes forget it
Beam goes okay
Beam goes bye
^C

If I grep that tail:

$ tail -f test.txt | grep Beam
Beam goes blah
Beam goes what?
Beam goes okay
Beam goes bye
^C

But if I awk that grep:

$ tail -f test.txt | grep Beam | awk '{print $3}'

Nothing no matter how long I wait. I suspect it's something to do with the way the stream works.

Anyone have any clue?

Answer 1

It's probably output buffering from grep. you can disable that with grep --line-buffered.

But you don't need to pipe output from grep into awk. awk can do regexp pattern matching all by itself.

tail -f test.txt | awk '/Beam/ {print $3}'

Answer 2

Using tail -f test.txt | awk '/Beam/{print $3}' works for me. As well as using tail -f test.txt | grep --line-buffered Beam | awk '{print $3}' (gnu grep).

The issue here is if awk received the data line-by-line or as one larger data block. The GNU version of grep sends output in larger blocks since it is more efficient, but awk needs to read line-by-line in order to output line-by-line.

Put this way: grep will only send data when the buffer is filled, awk is waiting for that buffer to be filled, so it's sending out nothing.

Answer 3

See the --line-buffered option of grep.