13

I am putting together a presentation for a non-technical audience. I have a program running in bash that outputs a continuous stream of values, a few of which are important. I would like to highlight the important results as they are displayed so the audience can get an idea of their frequency. The issue is that I can't get sed to operate on a running stream. It works fine if I put the results in a file, as in:

cat output.txt | sed "s/some text/some text bolded/"

But if I try the same thing on the running output, like this:

command | sed "s/some text/some text bolded/"

sed does nothing. Any thoughts?

As Lambert was helpful enough to point out, my saying that sed does nothing was vague. What is happening is that the program outputs to stdout (I'm pretty sure it's not writing to stderr) as it normally would, even if it's piped through sed.

The issue seems to be that the command calls a second program, which then outputs to stdout. There are a few lines printed by the first program; these I can edit. Then there is a stream of values printed by the second program; these I cannot edit.

Perl and awk methods do not work either.

  • 5
    Does stdbuf -o0 command | sed "s/some text/some text bolded/" work? – FloHimself May 4 '15 at 7:23
  • 3
    With 'sed does nothing' you mean that the substitution is not made, or don't you have any output? The command might be writing to stderr instead of stdout? If you want to highlight something you might use command|egrep 'some text|$' – Lambert May 4 '15 at 7:30
  • Is the command writing to stdout (and not to stderr)? – Anthon May 4 '15 at 7:30
  • 1
    Given that the text appears more than once you should add a g got "global" substitution, otherwise only the first occurrence on a line will be substituted: sed "s/old/new/g" – ph0t0nix May 4 '15 at 8:25
  • @ph0t0nix No, that isn't the issue, but it would have been an issue down the line. Thanks for the tip. – P Jones May 4 '15 at 16:58
12

Chances are that the command's output is buffered. When the command writes to a terminal, the buffer is flushed on every newline, so you see it appear at the expected rate. When the command writes to a pipe, the buffer is only flushed when it reaches a few kilobytes, so it lags a lot. Thus is the default behavior of the standard input/output library.

To force the command not to buffet its output, you can use unbuffer (from expect) or stdbuf (from GNU coreutils).

unbuffer command | sed …
stdbuf -o0 command | sed …
  • stdbuf did not work (it was mentioned previously, BTW), but unbuffer did!! You have no idea how happy you've made me. – P Jones May 5 '15 at 2:05
  • And thank you for the concise explanation. Very helpful. – P Jones May 5 '15 at 2:06
  • sed itself uses such a buffer, (cf ChennyStar post) so the examples here may not work since sed is the command to unbuffer: cat /etc/passwd | unbuffer sed but sed itself has an -u option, so grep might be more suiteable in this examples. Thanks a lot for your background information! Great answer! – math May 13 at 6:20
4

sed has an option for that :

-u, --unbuffered

Which loads minimal amounts of data from the input files and flushes the output buffers more often. See man sed for more details.

  • 1
    GNU sed only (not BSD sed), and I believe this still wouldn't prevent the buffering of the command at the start of the pipe. But good to mention it. :) – Wildcard Jan 20 '16 at 17:40
  • stdbuf manpage states that "If COMMAND adjusts the buffering of its standard streams ('tee' does for example) then that will override corresponding changes by 'stdbuf'." It looks like sed might fall into this category but the '-u' flag will make it useful in an unbuffered pipe chain. – MattSmith Feb 15 '18 at 21:57
1

The perl way:

command | perl -pe 's/(stuff)/\x1b[1m$1\x1b[0m/g'

or with a continous output:

A bash script for the output cont:

#!/bin/bash
while [ true ]
do
   echo "Some stuff"
done

Test with:

./cont | perl -pe 's/(stuff)/\x1b[1m${1}\x1b[0m/g'
  • \x1b[1m - bold or increased intensity
  • ${1} - the backreferenze
  • \x1b[0m - reset all attributes

Output:

Some stuff
Some stuff
Some stuff
Some stuff

More escape codes here.

2

I would use awk

  command | awk '/some important stuff/ { printf "%c[31m%s%c[0m\n",27,$0,27 ; next }
  { print ; } '

where

  • /some important stuff/ select important line, like in sed
  • printf "%c[31m%s%c[0m\n",27,$0,27 ; print in red
    • use 32,33 for green, yellow ...
    • $1, $2, can be use to select a specific field
  • other line are just printed 'as is'

the key point is that command should flush lines, but that should be the case if you have lot of output.

  • Note: The question does not say that the whole line should be "bolded", but just "some text". While possible to implement also that feature in awk (using sub() or gsub()), in case of this primitive substitution sed is certainly the appropriate tool. – Janis May 4 '15 at 10:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.