Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have multiple entries that describes an event in a very large log file, say A.log. I would like to do two things with the event entries in the log file:

  1. Count the number of occurrences of each such entry.(This is not a mandatory requirement but would be nice to have.)
  2. Extract the actual entries in a separate file and study them later on.

A typical event entry would look like the following and will have other texts between them. So in the example below there are two event entries, the first one containing two DataChangeEntry payload and second one containing one DataChangeEntry payload.

    Data control raising event :DataControl@263c015d[[
    #### DataChangeEvent #### on [DataControl name=PatternMatch_LegendTimeAxis, binding=.dynamicRegion1.                         beam_project_PatternMatch_dashboard_LegendTimeAxis_taskflow_LegendTimeAxis_beamDashboardLegendTimeAxisPageDef_beam_project_PatternMatch_dashboard_LegendTimeAxis_taskflow_LegendTimeAxis_beamDashboardLegendTimeAxis_xml_ps_taskflowid.dynamicRegion58.                                                                                                                         beam_project_PatternMatch_view_LegendTimeAxis_taskflow_LegendTimeAxis_beamVizLegendTimeAxisPageDef_beam_project_PatternMatch_view_LegendTimeAxis_taskflow_LegendTimeAxis_beamVizLegendTimeAxis_xml_ps_taskflowid.QueryIterator]
    Filter/Collection Id : 0
    Collection Level     : 0
    Sequence Id             : 616
    ViewSetId            : PatternMatch.LegendTimeAxis_V1_0_SN49
    ==== DataChangeEntry (#1)
    ChangeType           : UPDATE
    KeyPath              : [2014-06-26 06:15:00.0, 0]
    AttributeNames       : [DATAOBJECT_CREATED, COUNTX, QueryName]
    AttributeValues      : [2014-06-26 06:15:00.0, 11, StrAvgCallWaitTimeGreaterThanThreshold]
    AttributeTypes       : [java.sql.Timestamp, java.lang.Integer, java.lang.String,  ]
    ==== DataChangeEntry (#2)
    ChangeType           : UPDATE
    KeyPath              : [2014-06-26 06:15:00.0, 0]
    AttributeNames       : [DATAOBJECT_CREATED, COUNTX, QueryName]
    AttributeValues      : [2014-06-26 06:15:00.0, 9, AverageCallWaitingTimeGreateThanThreshold]
    AttributeTypes       : [java.sql.Timestamp, java.lang.Integer, java.lang.String,  ]

    ]]

someother non useful text
spanning multiple lines 

 Data control raising event :DataControl@263c015d[[
    #### DataChangeEvent #### on [DataControl name=PatternMatch_LegendTimeAxis, binding=.dynamicRegion1.                         beam_project_PatternMatch_dashboard_LegendTimeAxis_taskflow_LegendTimeAxis_beamDashboardLegendTimeAxisPageDef_beam_project_PatternMatch_dashboard_LegendTimeAxis_taskflow_LegendTimeAxis_beamDashboardLegendTimeAxis_xml_ps_taskflowid.dynamicRegion58.                                                                                                                         beam_project_PatternMatch_view_LegendTimeAxis_taskflow_LegendTimeAxis_beamVizLegendTimeAxisPageDef_beam_project_PatternMatch_view_LegendTimeAxis_taskflow_LegendTimeAxis_beamVizLegendTimeAxis_xml_ps_taskflowid.QueryIterator]
    Filter/Collection Id : 0
    Collection Level     : 0
    Sequence Id             : 616
    ViewSetId            : PatternMatch.LegendTimeAxis_V1_0_SN49
    ==== DataChangeEntry (#1)
    ChangeType           : UPDATE
    KeyPath              : [2014-06-26 06:15:00.0, 0]
    AttributeNames       : [DATAOBJECT_CREATED, COUNTX, QueryName]
    AttributeValues      : [2014-06-26 06:15:00.0, 11, StrAvgCallWaitTimeGreaterThanThreshold]
    AttributeTypes       : [java.sql.Timestamp, java.lang.Integer, java.lang.String,  ]

    ]]

Please note that the the number of ==== DataChangeEntry lines in an event entry can be variable. It can also be completely absent which would indicate empty events payload and is a error condition and would definitely like to catch this case as well.

Since in this case the output of entry spans across multiple lines I am not getting far using plain vanilla grep. So I am seeking expert advice.

P.S:

  1. Let me be more explicit about my requirement. I would like to capture the whole block of text shown above verbatim and optionally count the number of instances of such blocks encountered. The option to count the number of instances is good to have but not a mandatory requirement.
  2. If the solution to the problem is using awk I would like to save the awk file and re-use it. So please mention the steps to execute the script also. I know regex and grep but I am not familiar with sed and/or awk.
share|improve this question
    
Do they always start with Data control raising event ? –  LatinSuD Jun 26 at 16:28
    
@LatinSuD yes it always starts with that string. –  Geek Jun 26 at 16:28
    
I think this is a job for awk, using a "state machine" variable(s), but you should add some more information in order to get help with this, like exact tokens searched for and what you expect the end result to be. –  David Kohen Jun 26 at 16:42
    
@DavidKohen An event entry starts with "Data control raising event" token and ends in "]]" in a new line. I would like to find out each such event instances. –  Geek Jun 26 at 16:48
    
Find what about them? Count their amount? Print them all? Please edit your question and add sample expected output (with different sample inputs preferably). –  David Kohen Jun 26 at 16:55

2 Answers 2

This would do it i hope. Events go to events file. And messages go to stdout.

Save this file to myprogram.awk (for example):

#!/usr/bin/awk -f

BEGIN {
   s=0;  ### state. Active when parsing inside an event
   nevent=0;  ### Current event number
   printf "" > "events"
}

# Start of event
/^ *Data control raising event/ {
   s=1;
   dentries=0;
   print "*** Event number: " nevent >> "events"
   nevent++
}

# Standard event line
s==1 {
   print >> "events"
}

# DataChangeEntry line
/^ *==== DataChangeEntry/ {
   dentries ++
}

# End of event
s==1 && /^ *\]\]/ {
   s=0;
   print "" >> "events"
   if(dentries==0){
      print "Warning: Event " nevent " has no Data Entries"
   }
}

END {
   print "Total event count: " nevent
}

You can invoke it in different ways:

  • myprogram.awk inputfile.txt
  • awk -f myprogram.awk inputfile.txt

Sample output:

Warning: Event 3 has no Data Entries
Total event count: 3

You can check all the events together in the file called events in working directory.

share|improve this answer
    
You should increment the event counter separately from the event header (or have the operator before), this causes the header and footer to show different numbers and is less readable. –  David Kohen Jun 26 at 17:00
    
@LatinSuD I am not familiar with awk. So if you can add the part that I need to do to run the above program it will be very helpful. For me the input file is say A.log. –  Geek Jun 26 at 17:03
    
To use this script just replace the inputfile.txt with your file name, or better, remove the cat and pipe, and put your file name after the closing single quote. –  David Kohen Jun 26 at 17:09
    
@DavidKohen I would like to save this script. So if I save this as say findEvents.awk. Can I execute it like this: awk -f findEvents.awk A.log ? –  Geek Jun 26 at 17:18
    
You could, but you should save only the part within the single quotes in that file. –  David Kohen Jun 29 at 10:08

A very simple approach would be

awk '{print > NR".entry"}END{print NR" entries"}' RS="]]" file 

This will create a separate file for each entry and print the number of entries found to standard output.

Explanation

  • NR is the current line number in awk.
  • RS="]]" sets the record separator (what defines a "line") to ]]. This means that each entry will be treated as a single line by awk.
  • {print > NR".entry"} : this prints the current line (entry) into a file called [LineNumber].entry. So, 1.entry will contain the 1st, 2.entry the second and so on.
  • END{print NR" entries"} : the END block is executed after the entire input file has been processed. Therefore, at that point NR will be the number of entries processed.

You could save this as an alias or make it into a script like so:

#!/usr/bin/env bash
awk '{print > NR".entry"}END{print NR" entries"}' RS="]]" "$1"

You'd then run the script (assuming it's called foo.sh and is in your $PATH) with the target file as an argument:

foo.sh file

You can also tweak the output file names. For example, to have the files be called [date].[entry number].[entry] use this instead:

#!/usr/bin/env bash
date=$(date +%Y%m%d)
awk '{print > d"."NR".entry"}END{print NR" entries"}' RS="]]" d="$date" "$1"

The above assumes that your log file consists exclusively of "Event" entries. If that is not the case, and you can have other lines, and those lines should be ignored, use this instead:

 #!/usr/bin/env bash
date=$(date +%Y%m%d)
awk '{
        if(/\[\[/){a=1; c++;}
        if(/\]\]/){a=0; print > d"."c".entry"}
        if(a==1){print >> d"."c".entry"}
}' d="$date" file 

Or, as a one-liner:

awk '{if(/\[\[/){a=1; c++;}if(/\]\]/){a=0; print > d"."c".entry"}if(a==1){print >> d"."c".entry"}}' d=$(date +%Y%m%d) file 
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.