4

I have a logfile that contains many ^H and ^M characters, as the process that produces this file updates a text based progress bar.

When using cat the output is evaluated and appears human readable and concise. Below is an example output.

Epoch 11/120
4355/4355 [==============================] - ETA: 0s - loss: 0.0096   
Epoch 00011: val_loss did not improve from 0.00992
4355/4355 [==============================] - 1220s 280ms/step - loss: 0.0096 - val_loss: 0.0100

However, the file itself is huge compared to what the actual printed text from cat would suggest (70MB for around 900 lines) of text like above.

Below is a snippet of the actual text contained within the logfile.

1/Unknown - 0s 81us/step - loss: 0.5337^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^M  2/Unknown - 1s 438ms/step - loss: 0.5299^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^
H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^M      3/Unknown - 1s 386ms/step - loss: 0.5286^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^M      4/Unknown - 1s 357ms/step - loss: 0.5289^H^H^H^H^H^H^H^H^H^
H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^M      5/Unknown - 2s 339ms/step - loss: 0.5277^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^M      6/Unknown - 2s 327ms/
step - loss: 0.5258^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^M      7/Unknown - 2s 318ms/step - loss: 0.5250^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^
H^H^H^H^H^M      8/Unknown - 2s 312ms/step - loss: 0.5260^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^M      9/Unknown - 3s 307ms/step - loss: 0.5265^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^
H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^M     10/Unknown - 3s 303ms/step - loss: 0.5257^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^M     11/Unknown - 3s 299ms/step - loss: 0.5258^H^H^H^

Basically, I want to create a file that looks just like what cat produces.

Here's a few things i've tried with limited success:

  • tr -d '\b\r' < logfile > new_file removes all of the characters, but therefore leaves all of the undesired text.
  • cat logfile > new_file effectively just copies the file verbatim with no evaluation of the special characters.
  • cat logfile | col -b > new_file is pretty close, but does something odd on one of the repeated lines:
4355/4355 [==============================] - ETA: 0ss--loss::0.0096557
Epoch 00011: val_loss did not improve from 0.00992
4355/4355 [==============================] - 1220s 280ms/step - loss: 0.0096 - val_loss: 0.0100

Any help would be appreciated.

Thanks

5
  • 8
    "I want to create a file that looks just like what cat produces" – It's not what cat produces. It's what your terminal (terminal emulator) shows after it interprets ^H and such. What cat does is exactly like in cat logfile > new_file, except there's the terminal instead of new_file. Aug 13, 2021 at 11:44
  • @Quasímodo Good point, i've uploaded a sample of 20 lines from the logfile here drive.google.com/file/d/1KhZbEoFLQ0JCmPlw5rPKaVeWk-rAicq4/… hope that helps
    – Matt Lyon
    Aug 13, 2021 at 12:34
  • @rowboat your answer works perfectly, thank you! If you want to submit as an answer I will mark as the accepted answer.
    – Matt Lyon
    Aug 13, 2021 at 12:39
  • 1
    A general workaround may be to use an actual terminal emulator that allows you to save the content of its scrollback buffer. You cat the file to the terminal and save the ultimate result. Example for tmux: here. It seems "Save Output As…" in the "File" menu of konsole can do this. Other terminal emulators may provide similar functionality. Notes: (1) You need to make sure in advance the scrollback buffer is large enough. (2) I guess some terminal emulators may generate output with newlines where they wrap lines due to their width. Aug 13, 2021 at 13:13
  • 3
    It's really what col -b is for. You see a difference from when dumping the file to a terminal because backspace on your terminal when the cursor is at the beginning of the line happens to cause the cursor to move back to the end of the previous line, but try it on a terminal that doesn't such as the linux virtual consoles and you'll see the same (see the bw terminfo capability) Aug 13, 2021 at 15:24

2 Answers 2

4

Posting this as an answer for clarity.

As rowboat pointed out, in this case the command awk -F '\r' '{print $NF}' file works as intended, by removing everything after the last carriage return. Though this is not robust as zevzek pointed out.

I wrote a more robust solution in C++ below.

#include <fstream>
#include <string>
#include <iostream>

using namespace std;

string filter_string(string line, const char *bspace, const char *creturn){

    string new_str;

    for(string::size_type i = 0; i < line.size(); ++i) {
        // Step back if current string not empty
        if (line[i] == *bspace){
            if (new_str.size() != 0){
                new_str.pop_back();
            };
        // Reset on carriage return
        } else if (line[i] == *creturn){
            new_str = "";
        } else {
            new_str += line[i];
        };
    }

    return new_str;
};

int main(int argc, char* argv[]){
    const char backspace = '\x08';
    const char creturn = '\r';

    if (argc != 2){
        cerr << "USAGE: " << argv[0] << " [src]" << endl;
        return 1;
    }

    // Filter lines in file
    string line;
    ifstream infile(argv[1]);
    while (getline(infile, line)){
        cout << filter_string(line, &backspace, &creturn) << endl;
    };

    return 0;
};

Here each character in each line is iterated over, and if a ^H is present, the string is pushed back one (if not already empty), and if a ^M carriage return is present then the string is reset. The output is sent to stdout which can then be piped to a file.

1
  • 1
    const char newline = '\r'... yngh, that's the carriage return. Newline is \n. With the input data you have here, just removing up to the last CR of each line seems to be fine, since the backspaces are there just to clear the line before each time the cursor is moved back.
    – ilkkachu
    Aug 23, 2021 at 20:56
2

sed 's/.*\x0d//' logfile

Seems to do what you ask for.

Note that col -b fails because it is ignoring the whitespace:

$ echo $'--------\r1st try\r2nd   \r3rd\n' | col -b
3rd-try-

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .