33

I'm reading from a serial port connected to a gps device sending nmea strings.

A simplified invocation to illustrate my point:

  $ awk '{ print $0 }' /dev/ttyPSC9 
  GPGGA,073651.000,6310.1043,N,01436.1539,E,1,07,1.0,340.2,M,33.3,M,,0000*56
  $GPGSA,A,3,28,22,09,27,01,19,17,,,,,,2.3,1.0,2.0*39
  $GPRMC,073651.000,A,6310.1043,N,01436.1539,E,0.42,163.42,070312,,,A*67
  GPGGA,073652.000,6310.1043,N,01436.1540,E,1,07,1.0,339.2,M,33.3,M,,0000*55
  $GPGSA,A,3,28,22,09,27,01,19,17,,,,,,2.3,1.0,2.0*39

If I instead try to read from a pipe, awk buffers the input before sending it to stdout.

$ cat /dev/ttyPSC9 | awk '{ print $0 }'
<long pause>
GPGGA,073651.000,6310.1043,N,01436.1539,E,1,07,1.0,340.2,M,33.3,M,,0000*56
$GPGSA,A,3,28,22,09,27,01,19,17,,,,,,2.3,1.0,2.0*39
$GPRMC,073651.000,A,6310.1043,N,01436.1539,E,0.42,163.42,070312,,,A*67
GPGGA,073652.000,6310.1043,N,01436.1540,E,1,07,1.0,339.2,M,33.3,M,,0000*55
$GPGSA,A,3,28,22,09,27,01,19,17,,,,,,2.3,1.0,2.0*39

How can I avoid the buffering?

Edit: Kyle Jones suggested that cat is buffering it's output but that doesn't appear to be happening:

$ strace cat /dev/ttyPSC9 | awk '{ print $0 }'
write(1, "2,"..., 2)                    = 2
read(3, "E"..., 4096)                   = 1
write(1, "E"..., 1)                     = 1
read(3, ",0"..., 4096)                  = 2

When I think about it: I thought that a program used line buffering when writing to a terminal and "regular buffering" for all other cases. Then, why is cat not buffering more? Is the serial port signalling EOF? Then why is cat not terminated?

My awk is mawk 1.2.

2
  • 1
    BashFAQ 009 may be useful.
    – jw013
    Mar 7, 2012 at 8:21
  • @jw013: Thanks for the link, a great sum up of how buffering works in bash. Mar 7, 2012 at 8:31

3 Answers 3

61

I know it is an old question, but a one-liner may help those who come here searching:

cat /dev/ttyPSC9 | awk '{ print $0; system("")}'

system("") does the trick, and is POSIX compliant. Non-posix systems: beware.

There exists a more specific function fflush() that does the same, but is not available in older versions of awk.

An important piece of information from the docs regarding the use of system(""):

gawk treats this use of the system() function as a special case and is smart enough not to run a shell (or other command interpreter) with the empty command. Therefore, with gawk, this idiom is not only useful, it is also efficient.

3
  • This worked for me
    – redolent
    Jan 1, 2014 at 3:20
  • 7
    My awk doesn't do anything on neither fflush() nor system(""). My gawk honored it though. Jul 14, 2017 at 11:47
  • This worked on Mac's awk version 20200816 from Mac OS Ventura, thank god. May 9, 2023 at 18:57
15

It is likely to be buffering in awk, not cat. In the first case, awk believes it is interactive because its input and output are TTYs (even though they're different TTYs - I'm guessing that awk is not checking that). In the second, the input is a pipe so it runs non-interactively.

You will need to explicitly flush in your awk program. This is not portable, though.

For more background and details on how to flush output, read: http://www.gnu.org/software/gawk/manual/html_node/I_002fO-Functions.html

6
  • 12
    Thanks for the explanation. awk -W interactive '{print $0}' appears to do the trick. The 'W interactive option is available on my awk version (mawk 1.2) but I dunno if it's a standard option. Mar 7, 2012 at 8:29
  • 3
    @dannas -W is not in the POSIX standard for awk. I'm not sure what to do if you need maximum portability.
    – jw013
    Mar 7, 2012 at 8:46
  • I'm accepting this answer since it explains why awk is doing full buffering in my example, instead of line buffering - it checks the if the input is a tty as well as the output. I only thought that it would check the output. Mar 7, 2012 at 8:57
  • @jw013: Thanks for looking up the standard. For me, I just wanted to understand why awk was doing full buffering and I think I do now. Mar 7, 2012 at 9:00
  • @dannas I can confirm that -W interactive is at least supported in the Ubuntu 12.04 (and presumably newer) distribution of awk, which is mawk.
    – Jason C
    Jun 11, 2014 at 19:38
5

Old topic but maybe worth to add one more solution which changes stream buffering behavior in a transparent way without some magic like system("") :-)

cat /dev/ttyPSC9 | stdbuf --output=L awk '{print $0}'

Used it myself recently to catch a D-Bus event.

gdbus monitor -y -d org.freedesktop.login1 | stdbuf -oL  grep LockedHint
1
  • Thanks, this fixed my problem on RHEL 7.
    – Mark Booth
    Jan 17, 2023 at 15:40

Not the answer you're looking for? Browse other questions tagged .