Redirecting output that spams terminal with carriage returns

Question

I have a command that creates very verbose output, on the order of hundreds of lines per second. However, the command uses \r to overwrite the previous line of output, in a manner similar to a progress bar. Occasionally it writes a newline to the terminal, which "bakes" the current output line.

When redirecting this output to a file, I get hundreds of megs of output - each line is written to the file, instead of being 'overwritten' when the carriage return occurs.

I understand this is the expected behavior, and that one way to solve it would be to make the program smarter, and realize it's being redirected to the file and not print this interactive status. However, I can't modify this program.

Is there some way I can pipe/filter this output so that what ends up in the final output file is the same as what I would see if I ran it interactively on the terminal?

I've tried:

spammy_cr_command | uniq

... which outputs the same as without uniq

and also:

spammy_cr_command | sed '/\r/d'

... which deletes the "baked" lines that contain the newline character as well.

Answer 1

cmd | sed -e 's/.*\r//' > file

This will replace all text on each line that is followed by a carriage return with nothing, leaving only the part of the line after the final carriage return behind. This is not necessarily the same as what would be left on the terminal, though, but it's a close approximation most of the time.

---

In particular, the case where a line is longer than its successor isn't handled. This program would give incorrect results:

printf 'abcdefg\rxyz\n'
printf '123456789\r\nxyz\n'

because what would be left behind visibly is

xyzdefg
123456789
xyz

but the sed would skip all the unerased characters as well and give

xyz

xyz

You can determine whether your program behaves like that or not. It's not uncommon for progress bars and the like to rest the cursor on the left-hand edge, which may not give the result you wanted.

Answer 2

For very primitive TTY-37 output, the col command solves this without the problems of sed mentioned in M. Homer's answer. (For output that is not simple TTY-37 output, and contains terminal escape and control sequences, neither col nor sed are the tool for the job; but Stack Exchange has had a Q&A on that for almost eight years already.)

% (
printf 'abcdefg\rxyz\n'
printf '123456789\r\nxyz\n'
) | col -b
xyzdefg
123456789
xyz
%

Answer 3

Something closer to the overwriting behaviour can be done with GNU awk:

BEGIN { 
  RS = "[\r\n]"                   # split records on either CR or LF
  a = ""                          # variable to save the text for overwriting
} 
{
  a = $0 substr(a, 1 + length)    # save current line, add trailing part of saved text
} 
RT ~ /\n/ {                       # LF, time to print and reset
  print a;
  a = ""
}

Using Michael Homer's example:

~ awk 'BEGIN { RS="[\r\n]" } {a = $0 substr(a, 1 + length)} RT ~ /\n/ {print a; a=""}' foo
xyzdefg
123456789
xyz

GNU awk is needed for the RT variable, which contains the record separator text that matched the RS regex for that record.