How to use sed to manipulate continuously streaming output?

Question

I am putting together a presentation for a non-technical audience. I have a program running in bash that outputs a continuous stream of values, a few of which are important. I would like to highlight the important results as they are displayed so the audience can get an idea of their frequency. The issue is that I can't get sed to operate on a running stream. It works fine if I put the results in a file, as in:

cat output.txt | sed "s/some text/some text bolded/"

But if I try the same thing on the running output, like this:

command | sed "s/some text/some text bolded/"

sed does nothing. Any thoughts?

As Lambert was helpful enough to point out, my saying that sed does nothing was vague. What is happening is that the program outputs to stdout (I'm pretty sure it's not writing to stderr) as it normally would, even if it's piped through sed.

The issue seems to be that the command calls a second program, which then outputs to stdout. There are a few lines printed by the first program; these I can edit. Then there is a stream of values printed by the second program; these I cannot edit.

Perl and awk methods do not work either.

Answer 1

Chances are that the command's output is buffered. When the command writes to a terminal, the buffer is flushed on every newline, so you see it appear at the expected rate. When the command writes to a pipe, the buffer is only flushed when it reaches a few kilobytes, so it lags a lot. Thus is the default behavior of the standard input/output library.

To force the command not to buffet its output, you can use unbuffer (from expect) or stdbuf (from GNU coreutils).

unbuffer command | sed …
stdbuf -o0 command | sed …

Answer 2

sed has an option for that :

-u, --unbuffered

Which loads minimal amounts of data from the input files and flushes the output buffers more often. See man sed for more details.

Answer 3

I would use awk

  command | awk '/some important stuff/ { printf "%c[31m%s%c[0m\n",27,$0,27 ; next }
  { print ; } '

where

• /some important stuff/ select important line, like in sed
• printf "%c[31m%s%c[0m\n",27,$0,27 ; print in red
  • use 32,33 for green, yellow ...
  • $1, $2, can be use to select a specific field
• other line are just printed 'as is'

the key point is that command should flush lines, but that should be the case if you have lot of output.

Answer 4

The perl way:

command | perl -pe 's/(stuff)/\x1b[1m$1\x1b[0m/g'

or with a continous output:

A bash script for the output cont:

#!/bin/bash
while [ true ]
do
   echo "Some stuff"
done

Test with:

./cont | perl -pe 's/(stuff)/\x1b[1m${1}\x1b[0m/g'

• \x1b[1m - bold or increased intensity
• ${1} - the backreferenze
• \x1b[0m - reset all attributes

Output:

Some stuff
Some stuff
Some stuff
Some stuff

More escape codes here.