0

I have a text file with output like this:

file_0108.json
2023-02-22T01:15:05.531+0000    connected to: mongodb://[**REDACTED**]@localhost
2023-02-22T01:15:08.531+0000    [######..................] db.coll  64.7MB/255MB (25.4%)
2023-02-22T01:15:11.531+0000    [############............] db.coll  128MB/255MB (50.3%)
2023-02-22T01:15:14.531+0000    [##################......] db.coll  196MB/255MB (76.9%)
2023-02-22T01:15:17.286+0000    [########################] db.coll  255MB/255MB (100.0%)
2023-02-22T01:15:17.286+0000    380757 document(s) imported successfully. 0 document(s) failed to import.

The file numbers (start of the block go from 0000 - 1000. Not all of the files imported successfully. How can I find every block of text in the file that starts with the file name, and ends with:

xxxxx document(s) imported successfully. 0 document(s) failed to import

then delete them, leaving only the errors?
Each block could have a different number of lines between the file name and the end of the block.
There are blocks with errors, but the errors can be different, so I thought it'd be easier to just remove the ones without errors.

Sample error block:

file_0293.json  
2023-02-22T01:52:15.303+0000    connected to: mongodb://[**REDACTED**]@localhost  
2023-02-22T01:52:16.836+0000    Failed: error processing document #46401: invalid character ',' after object key  
2023-02-22T01:52:16.836+0000    46000 document(s) imported successfully. 0 document(s) failed to import.
Improve this question
8
  • If we remove everything between filename and …failed to import, that would also remove all blocks with errors. And it's not helpful if the last line contains "0 document(s) failed to import". So how would we know an error occured? Search for "Failed:" or "error" in those blocks?
    – Freddy
    Feb 22 at 15:46
  • 1
    Please give us some more details, maybe a few more examples? Are all files in the format of file_NNNN.json or can some not start with file_ or not end with .json or any other change? Are the lines with the file name the only lines that have just one field and no whitespace? Can you have whitespace in filenames?
    – terdon ♦
    Feb 22 at 16:36
  • @terdon The output is from running a bash for loop that executes echo <filename> && mongoimport. The examples are representative, as you said, one line with just the file's name, then a variable number of lines from the output of the mongoimport command. The files names are all in the format file_NNNN.json.
    – Sebastian
    Feb 22 at 16:41
  • 2
    And in the case of an error, you still have 0 document(s) failed to import.? Shouldn't you have N document(s) failed to import. where N>0?
    – terdon ♦
    Feb 22 at 16:44
  • @terdon I was hoping so, but mongo is an odd duck sometimes. My interpretation of the output is that it imported 46k documents, then stopped trying. Each of these files had ~380k documents. So it didn't finish, but it said 0 failed. My only guess is that it stopped trying after the error.
    – Sebastian
    Feb 22 at 20:07

4 Answers 4

Reset to default
0

If there are no blank lines inside each block of text, then you could use sed to insert a blank line after each line with imported successfully and then process the file in "paragraphs" (blocks of text separated by one or more blank lines). For example:

sed -e $'/imported successfully/a\\\n' filename |
  perl -00 -n -e 'print if /Failed:/'

Also, you mentioned in a comment that your input file is generated by a bash for loop running echo <filename> && mongoimport. I suggest you change that to run echo <filename> && mongoimport ; echo so that future runs will already have their output separated into paragraphs. sed would no longer be needed to insert a newline, so you could just run:

perl -00 -n -e 'print if /Failed:/' filename
Improve this answer
2
  • Good point on the extra echo in the for loop. Thanks.
    – Sebastian
    Feb 23 at 14:48
  • You might even consider writing each file_0108.json block into its own 0108-file.log. Then you can combine all files without error into one combined logfile later, those with error into an other.
    – user unknown
    Feb 23 at 21:53
0

I tried with the following text file with output,

file_0108.json
2023-02-22T01:15:05.531+0000    connected to: mongodb://[**REDACTED**]@localhost
2023-02-22T01:15:08.531+0000    [######..................] db.coll  64.7MB/255MB (25.4%)
2023-02-22T01:15:11.531+0000    [############............] db.coll  128MB/255MB (50.3%)
2023-02-22T01:15:14.531+0000    [##################......] db.coll  196MB/255MB (76.9%)
2023-02-22T01:15:17.286+0000    [########################] db.coll  255MB/255MB (100.0%)
2023-02-22T01:15:17.286+0000    380757 document(s) imported successfully. 0 document(s) failed to import.
file_0293.json  
2023-02-22T01:52:15.303+0000    connected to: mongodb://[**REDACTED**]@localhost  
2023-02-22T01:52:16.836+0000    Failed: error processing document #46401: invalid character ',' after object key  
2023-02-22T01:52:16.836+0000    Failed: error processing document #46427: invalid character ',' after object key  
2023-02-22T01:52:16.836+0000    46000 document(s) imported successfully. 0 document(s) failed to import.

and the following command line produced what I think is useful output to the terminal.

$ grep -e 'file_.*\.json' -e 'Failed:' file.txt | sed 's/json/json:/'|grep -B1 'Failed:'
file_0293.json:  
2023-02-22T01:52:16.836+0000    Failed: error processing document #46401: invalid character ',' after object key  
2023-02-22T01:52:16.836+0000    Failed: error processing document #46427: invalid character ',' after object key

You can redirect it to a file if you wish for example like this to make sure that you print both output to the standard output and the error output, ... > errors.txt 2>&1,

grep -e 'file_.*\.json' -e 'Failed:' file.txt | sed 's/json/json:/'|grep -B1 'Failed:' > errors.txt 2>&1
Improve this answer
0

Using awk:

awk -v startblock='^file_[0-9][0-9][0-9][0-9]\\.json$' \
    -v endblock='document\\(s\\) failed to import\\.$' '
    $0 ~ startblock {
        error=0
        s=""
    }
    {
        s=(s=="" ? "" : s ORS) $0
    }
    $0 ~ endblock && (error || $0 !~ " 0 " endblock) {
        print s
        next
    }
    tolower($0) ~ /failed|error|invalid/ {
        error=1
    }
' file

This prints all blocks containing a case-insensitive match of failed, error or invalid between start and end of block or where the end of the block line contains n document(s) failed to import. where n is non-zero.

Improve this answer
0

To do what you asked for using any awk would be:

awk '
    /^file_[0-9]+\.json$/ {
        printf "%s", rec
        rec = ""
    }
    { rec = rec $0 ORS }
    /document\(s) imported successfully. 0 document\(s) failed to import/ {
        rec = ""
    }
    END { printf "%s", rec }
' file

but your posted sample input doesn't match what you asked for. I THINK what you probably really want is something like (again using any awk):

awk '
    /^file_[0-9]+\.json$/ {
        if ( !bad ) printf "%s", rec
        rec = bad = ""
    }
    /Failed:/ { bad = 1 }
    { rec = rec $0 ORS }
    END { if ( !bad ) printf "%s", rec }
' file
Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .