how to properly log the output of a console program that frequently updates "parts" of the screen, resulting in a messy log file?

Question

I have some Java executable (jar) that is run my some shell script from a cron job once every night. That executable does not print log statements "as usual" just by printing them out in a sequential manner like line after line (print after print), but while it processes its data its printing a single line with status data and then "overwrite" or "update" just that single line over and over again, until its done with this part of processing.

Examples:

A common program would output something line this:

Program XYZ started..<cr+lf>
processing 1<cr+lf>
processing 2<cr+lf>
processing 3<cr+lf>
im done<cr+lf>

Simple! I can easily redirect its output to a file and I'm done.

But the program I will have to deal with does it more like this:

Program XYZ started..<cr+lf>
processing 1<cr>
processing 1 - part a<cr>
processing 1 - part b<cr>
processing 1 - part c<cr>
processing 1 - part d<cr>
<cr+lf>
processing 2<cr>
processing 2 - part a<cr>
processing 2 - part b<cr>
processing 2 - part c<cr>
processing 2 - part d<cr>
<cr+lf>
processing 3<cr+lf>
im done<cr+lf>

but much more intensive than I showed here. So it just not always overwrite status lines, but does it a lot while processing. So just redirecting or appending its output to a log file does not seem to make a lot of sense here, since it will just clutter the log file with millions of screen-updates that can not be viewed and understood that easily by humans.

So my question is: is there a way or tool for programs that may behave like that to "log" or maybe "screenshot every 30 seconds" their output to a log file?

Answer 1

Trivially, assuming there are no vt100 escapes to handle, you might try pushing the output through

sed 's/\r$//; s/.*\r//'

or the awk equivalent

awk '{ sub("\r$",""); sub(".*\r",""); print}'

but this assumes your sed or awk can handle very long lines, as the carriage-returns obviously aren't newlines. Also, carriage-return only moves the cursor to the start of the line, but doesn't erase what is on that line, so strictly you should keep the long line before \r\r\n in your example.

To cope with the effectively long lines you could use tr '\r\n' '\n\001' to translate the \r to newline, and simultaneously the newlines to some other character not in your data, like control-a.

Or you can change awk's input record separator with RS='\r', if your awk allows it. This gives you a script like:

awk -v RS='\r' '
{ if(sub("\n","")){print last;last=""}
  if($0!="")last = $0
}'

Answer 2

I use either script2log (simple), or vile-pager (vi like emacs) for this sort of thing. The script handles overstrikes of the sort mentioned, as well as backspace/overstriking. To handle cursor-movement (as you would get in editing a command-line), a simple script like this is not adequate. The script removes ANSI-style escape sequences, however.

The script uses sed. Here is just the sed-command:

        # Trim ordinary ANSI sequences, then OSC sequences, then backspace
        # sequences, then trailing CR's and finally overstruck sections of
        # lines.
        #
        # There are still several interesting cases which cannot be handled
        # with a script of this sort.  For example:
        #       CSI K (clear line)
        #       cursor movement within the line
        sed \
                -e 's/[[][?]\{0,1\}[;0-9]*[@-~]//g' \
                -e 's/[]][^]*//g' \
                -e 's/[]][^]*\\//g' \
                -e ':loop; s/[^]\(.\)/\1/g; t loop;' \
                -e 's/ *$//g' \
                -e 's/^.* //g' \
                -e 's/[^[]//g'

which contains literal escape characters rather than rely upon non-POSIX extensions.

Further reading:

• sed - stream editor (POSIX)
• ECMA-48: Control Functions for Coded Character Sets