Evaluate large file with ^H and ^M characters

Question

I have a logfile that contains many ^H and ^M characters, as the process that produces this file updates a text based progress bar.

When using cat the output is evaluated and appears human readable and concise. Below is an example output.

Epoch 11/120
4355/4355 [==============================] - ETA: 0s - loss: 0.0096   
Epoch 00011: val_loss did not improve from 0.00992
4355/4355 [==============================] - 1220s 280ms/step - loss: 0.0096 - val_loss: 0.0100

However, the file itself is huge compared to what the actual printed text from cat would suggest (70MB for around 900 lines) of text like above.

Below is a snippet of the actual text contained within the logfile.

1/Unknown - 0s 81us/step - loss: 0.5337^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^M  2/Unknown - 1s 438ms/step - loss: 0.5299^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^
H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^M      3/Unknown - 1s 386ms/step - loss: 0.5286^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^M      4/Unknown - 1s 357ms/step - loss: 0.5289^H^H^H^H^H^H^H^H^H^
H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^M      5/Unknown - 2s 339ms/step - loss: 0.5277^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^M      6/Unknown - 2s 327ms/
step - loss: 0.5258^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^M      7/Unknown - 2s 318ms/step - loss: 0.5250^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^
H^H^H^H^H^M      8/Unknown - 2s 312ms/step - loss: 0.5260^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^M      9/Unknown - 3s 307ms/step - loss: 0.5265^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^
H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^M     10/Unknown - 3s 303ms/step - loss: 0.5257^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^M     11/Unknown - 3s 299ms/step - loss: 0.5258^H^H^H^

Basically, I want to create a file that looks just like what cat produces.

Here's a few things i've tried with limited success:

• tr -d '\b\r' < logfile > new_file removes all of the characters, but therefore leaves all of the undesired text.
• cat logfile > new_file effectively just copies the file verbatim with no evaluation of the special characters.
• cat logfile | col -b > new_file is pretty close, but does something odd on one of the repeated lines:

4355/4355 [==============================] - ETA: 0ss--loss::0.0096557
Epoch 00011: val_loss did not improve from 0.00992
4355/4355 [==============================] - 1220s 280ms/step - loss: 0.0096 - val_loss: 0.0100

Any help would be appreciated.

Thanks

Answer 1

Posting this as an answer for clarity.

As rowboat pointed out, in this case the command awk -F '\r' '{print $NF}' file works as intended, by removing everything after the last carriage return. Though this is not robust as zevzek pointed out.

I wrote a more robust solution in C++ below.

#include <fstream>
#include <string>
#include <iostream>

using namespace std;

string filter_string(string line, const char *bspace, const char *creturn){

    string new_str;

    for(string::size_type i = 0; i < line.size(); ++i) {
        // Step back if current string not empty
        if (line[i] == *bspace){
            if (new_str.size() != 0){
                new_str.pop_back();
            };
        // Reset on carriage return
        } else if (line[i] == *creturn){
            new_str = "";
        } else {
            new_str += line[i];
        };
    }

    return new_str;
};

int main(int argc, char* argv[]){
    const char backspace = '\x08';
    const char creturn = '\r';

    if (argc != 2){
        cerr << "USAGE: " << argv[0] << " [src]" << endl;
        return 1;
    }

    // Filter lines in file
    string line;
    ifstream infile(argv[1]);
    while (getline(infile, line)){
        cout << filter_string(line, &backspace, &creturn) << endl;
    };

    return 0;
};

Here each character in each line is iterated over, and if a ^H is present, the string is pushed back one (if not already empty), and if a ^M carriage return is present then the string is reset. The output is sent to stdout which can then be piped to a file.

Answer 2

sed 's/.*\x0d//' logfile

Seems to do what you ask for.

Note that col -b fails because it is ignoring the whitespace:

$ echo $'--------\r1st try\r2nd   \r3rd\n' | col -b
3rd-try-