Grepping for a block of text with parts that can be optional

Question

I have multiple entries that describes an event in a very large log file, say A.log. I would like to do two things with the event entries in the log file:

1. Count the number of occurrences of each such entry.(This is not a mandatory requirement but would be nice to have.)
2. Extract the actual entries in a separate file and study them later on.

A typical event entry would look like the following and will have other texts between them. So in the example below there are two event entries, the first one containing two DataChangeEntry payload and second one containing one DataChangeEntry payload.

    Data control raising event :DataControl@263c015d[[
    #### DataChangeEvent #### on [DataControl name=PatternMatch_LegendTimeAxis, binding=.dynamicRegion1.                         beam_project_PatternMatch_dashboard_LegendTimeAxis_taskflow_LegendTimeAxis_beamDashboardLegendTimeAxisPageDef_beam_project_PatternMatch_dashboard_LegendTimeAxis_taskflow_LegendTimeAxis_beamDashboardLegendTimeAxis_xml_ps_taskflowid.dynamicRegion58.                                                                                                                         beam_project_PatternMatch_view_LegendTimeAxis_taskflow_LegendTimeAxis_beamVizLegendTimeAxisPageDef_beam_project_PatternMatch_view_LegendTimeAxis_taskflow_LegendTimeAxis_beamVizLegendTimeAxis_xml_ps_taskflowid.QueryIterator]
    Filter/Collection Id : 0
    Collection Level     : 0
    Sequence Id             : 616
    ViewSetId            : PatternMatch.LegendTimeAxis_V1_0_SN49
    ==== DataChangeEntry (#1)
    ChangeType           : UPDATE
    KeyPath              : [2014-06-26 06:15:00.0, 0]
    AttributeNames       : [DATAOBJECT_CREATED, COUNTX, QueryName]
    AttributeValues      : [2014-06-26 06:15:00.0, 11, StrAvgCallWaitTimeGreaterThanThreshold]
    AttributeTypes       : [java.sql.Timestamp, java.lang.Integer, java.lang.String,  ]
    ==== DataChangeEntry (#2)
    ChangeType           : UPDATE
    KeyPath              : [2014-06-26 06:15:00.0, 0]
    AttributeNames       : [DATAOBJECT_CREATED, COUNTX, QueryName]
    AttributeValues      : [2014-06-26 06:15:00.0, 9, AverageCallWaitingTimeGreateThanThreshold]
    AttributeTypes       : [java.sql.Timestamp, java.lang.Integer, java.lang.String,  ]

    ]]

someother non useful text
spanning multiple lines 

 Data control raising event :DataControl@263c015d[[
    #### DataChangeEvent #### on [DataControl name=PatternMatch_LegendTimeAxis, binding=.dynamicRegion1.                         beam_project_PatternMatch_dashboard_LegendTimeAxis_taskflow_LegendTimeAxis_beamDashboardLegendTimeAxisPageDef_beam_project_PatternMatch_dashboard_LegendTimeAxis_taskflow_LegendTimeAxis_beamDashboardLegendTimeAxis_xml_ps_taskflowid.dynamicRegion58.                                                                                                                         beam_project_PatternMatch_view_LegendTimeAxis_taskflow_LegendTimeAxis_beamVizLegendTimeAxisPageDef_beam_project_PatternMatch_view_LegendTimeAxis_taskflow_LegendTimeAxis_beamVizLegendTimeAxis_xml_ps_taskflowid.QueryIterator]
    Filter/Collection Id : 0
    Collection Level     : 0
    Sequence Id             : 616
    ViewSetId            : PatternMatch.LegendTimeAxis_V1_0_SN49
    ==== DataChangeEntry (#1)
    ChangeType           : UPDATE
    KeyPath              : [2014-06-26 06:15:00.0, 0]
    AttributeNames       : [DATAOBJECT_CREATED, COUNTX, QueryName]
    AttributeValues      : [2014-06-26 06:15:00.0, 11, StrAvgCallWaitTimeGreaterThanThreshold]
    AttributeTypes       : [java.sql.Timestamp, java.lang.Integer, java.lang.String,  ]

    ]]

Please note that the the number of ==== DataChangeEntry lines in an event entry can be variable. It can also be completely absent which would indicate empty events payload and is a error condition and would definitely like to catch this case as well.

Since in this case the output of entry spans across multiple lines I am not getting far using plain vanilla grep. So I am seeking expert advice.

P.S:

1. Let me be more explicit about my requirement. I would like to capture the whole block of text shown above verbatim and optionally count the number of instances of such blocks encountered. The option to count the number of instances is good to have but not a mandatory requirement.
2. If the solution to the problem is using awk I would like to save the awk file and re-use it. So please mention the steps to execute the script also. I know regex and grep but I am not familiar with sed and/or awk.

Answer 1

This would do it i hope. Events go to events file. And messages go to stdout.

Save this file to myprogram.awk (for example):

#!/usr/bin/awk -f

BEGIN {
   s=0;  ### state. Active when parsing inside an event
   nevent=0;  ### Current event number
   printf "" > "events"
}

# Start of event
/^ *Data control raising event/ {
   s=1;
   dentries=0;
   print "*** Event number: " nevent >> "events"
   nevent++
}

# Standard event line
s==1 {
   print >> "events"
}

# DataChangeEntry line
/^ *==== DataChangeEntry/ {
   dentries ++
}

# End of event
s==1 && /^ *\]\]/ {
   s=0;
   print "" >> "events"
   if(dentries==0){
      print "Warning: Event " nevent " has no Data Entries"
   }
}

END {
   print "Total event count: " nevent
}

You can invoke it in different ways:

• myprogram.awk inputfile.txt
• awk -f myprogram.awk inputfile.txt

Sample output:

Warning: Event 3 has no Data Entries
Total event count: 3

You can check all the events together in the file called events in working directory.

Answer 2

A very simple approach would be

awk '{print > NR".entry"}END{print NR" entries"}' RS="]]" file 

This will create a separate file for each entry and print the number of entries found to standard output.

Explanation

• NR is the current line number in awk.
• RS="]]" sets the record separator (what defines a "line") to ]]. This means that each entry will be treated as a single line by awk.
• {print > NR".entry"} : this prints the current line (entry) into a file called [LineNumber].entry. So, 1.entry will contain the 1st, 2.entry the second and so on.
• END{print NR" entries"} : the END block is executed after the entire input file has been processed. Therefore, at that point NR will be the number of entries processed.

You could save this as an alias or make it into a script like so:

#!/usr/bin/env bash
awk '{print > NR".entry"}END{print NR" entries"}' RS="]]" "$1"

You'd then run the script (assuming it's called foo.sh and is in your $PATH) with the target file as an argument:

foo.sh file

You can also tweak the output file names. For example, to have the files be called [date].[entry number].[entry] use this instead:

#!/usr/bin/env bash
date=$(date +%Y%m%d)
awk '{print > d"."NR".entry"}END{print NR" entries"}' RS="]]" d="$date" "$1"

---

The above assumes that your log file consists exclusively of "Event" entries. If that is not the case, and you can have other lines, and those lines should be ignored, use this instead:

 #!/usr/bin/env bash
date=$(date +%Y%m%d)
awk '{
        if(/\[\[/){a=1; c++;}
        if(/\]\]/){a=0; print > d"."c".entry"}
        if(a==1){print >> d"."c".entry"}
}' d="$date" file 

Or, as a one-liner:

awk '{if(/\[\[/){a=1; c++;}if(/\]\]/){a=0; print > d"."c".entry"}if(a==1){print >> d"."c".entry"}}' d=$(date +%Y%m%d) file