Grabbing chunks of log files that have multiple matching criteria, but the matching criteria is not on the same line

Question

code:

grep -rI --exclude=*_*.log -B20 -A1 "Status:   Error" > /var/log/alertError.log

Input log example:

[06/07/20 20:38:53.911]:loopback ST:                  token-src-name() 

[06/07/20 20:38:53.914]:loopback ST:                    Token Value: "DVADER". 

[06/07/20 20:38:53.916]:loopback ST:                  token-text(",OU=users,O=data") 

[06/07/20 20:38:53.919]:loopback ST:    Arg Value: "CN=DVADER,OU=users,O=data". 

[06/07/20 20:38:53.922]:loopback ST:                description("Removed by Termination Process") 

[06/07/20 20:38:53.926]:loopback ST:             token-text("Removed by Termination Process") 

[06/07/20 20:38:53.929]:loopback ST:                  Arg Value: "Removed by Termination Process". 

[06/07/20 20:38:53.943]:loopback ST: DirXML Log Event -------------------

     Driver:   \StarWars\system\Driver Set\User Processor

     Channel:  Subscriber

     Status:   Error

     Message:  Code(-9217) Error in

This input log shows a section of the log I want to grab. However I want to match on the date for the days I want to search the log. I am using grep as I can do a recursive search of all log files in a structure. My grep returns all of the data I want, except I now want to exclude chunks of code that is older. So the date is not on all lines. The grep is returning a chunk of code with the switches B and A that I need. So if you saw the entire line, the message and status lines are consistent with multiple operations going on. I could grab all types of Message or status values with this grep command, but I just don't know how to then eliminate all of the results to only those chunks of code that are for a given date range.

Answer 1

Here is the python solution:

with open("log.txt") as f:                                  # open file log.txt
    lines = f.readlines()                                   # load lines
    nonempty = filter(lambda x: x.strip() != "", lines)     # filter out empty lines
    newlines = []                                           # list for out result lines
    for l in nonempty:                                      # iterate lines
        l = l.rstrip()                                      # cut '\n' from the right of each line
        last_idx = len(newlines) - 1                        # index of the last element in the list
        if l.startswith(" "):                               # this lines are lines with your traceback
            newlines[last_idx] += l                         # add to the "normal" log element
        else:                                               # this are "normal" elements
            newlines.append(l)                              # add them to the list

    print("\n".join(newlines))                              # create output and print to stdout

This output contains "Status" and "Date" on the same line and you could grep it

Place it (say normalize.py) near you log file (say log.txt) and run with python3 normalize.py

Answer 2

Your question isn't clear but is this what you're trying to do?

$ awk -v tgt='06/07/20' '
    /^\[/ { prt() }
    NF { rec = rec $0 ORS }
    END { prt() }

    function prt() {
        if ( index(rec,tgt) == 2 ) {
            printf "%s", rec
        }
        rec = ""
    }
' file
[06/07/20 20:38:53.911]:loopback ST:                  token-src-name()
[06/07/20 20:38:53.914]:loopback ST:                    Token Value: "DVADER".
[06/07/20 20:38:53.916]:loopback ST:                  token-text(",OU=users,O=data")
[06/07/20 20:38:53.919]:loopback ST:    Arg Value: "CN=DVADER,OU=users,O=data".
[06/07/20 20:38:53.922]:loopback ST:                description("Removed by Termination Process")
[06/07/20 20:38:53.926]:loopback ST:             token-text("Removed by Termination Process")
[06/07/20 20:38:53.929]:loopback ST:                  Arg Value: "Removed by Termination Process".
[06/07/20 20:38:53.943]:loopback ST: DirXML Log Event -------------------
     Driver:   \StarWars\system\Driver Set\User Processor
     Channel:  Subscriber
     Status:   Error
     Message:  Code(-9217) Error in

or maybe this?

$ awk -v tgt='06/07/20' '
    /^\[/ { prt() }
    NF { rec = rec $0 ORS }
    END { prt() }

    function prt() {
        if ( (index(rec,tgt) == 2) && (rec ~ /Status:[[:space:]]+Error/) ) {
            printf "%s", rec
        }
        rec = ""
    }
' file
[06/07/20 20:38:53.943]:loopback ST: DirXML Log Event -------------------
     Driver:   \StarWars\system\Driver Set\User Processor
     Channel:  Subscriber
     Status:   Error
     Message:  Code(-9217) Error in

You can call awk from find easily, e.g.:

find . -type f -name '*.log' -exec awk '....' {} +