4

I have a text file (events.dat) like this below (note only an extract is shown)

RepID12 01/01/2010 20:56:00 S10
RepID12 01/01/2010 20:56:00 S03
RepID20 01/01/2010 20:56:00 S17
RepID33 01/01/2010 20:56:00 S02
RepID33 01/01/2010 20:56:00 S18
RepID38 01/01/2010 20:56:00 S11
RepID39 01/01/2010 20:56:00 S20
RepID26 02/01/2010 01:39:00 S20
RepID29 02/01/2010 01:39:00 S16
RepID29 02/01/2010 01:39:00 S03
RepID22 02/01/2010 01:39:09 S01
RepID26 02/01/2010 01:39:09 S02
RepID40 02/01/2010 01:39:18 S02
RepID38 02/01/2010 01:39:09 S05
RepID31 02/01/2010 01:39:09 S06
RepID31 02/01/2010 01:39:09 S08
RepID09 02/01/2010 01:39:09 S09
RepID23 02/01/2010 01:39:18 S09
RepID19 02/01/2010 01:40:09 S09
RepID21 02/01/2010 01:40:18 S09
RepID28 02/01/2010 01:40:27 S09
RepID43 02/01/2010 01:40:09 S14

and so on, covering events for a total time span of 48 hours. I'd like to print out only the lines when more than say 60 events per minute are found.

For example with this command I can count how many events there are in 1 specific minute:

grep "02/01/2010 01:39" events.dat | wc -l

and this will return 60 (for example) that should be the max number of events/min.

How can I do the same but checking each minute for the whole 48 hours and print out only the lines where more than 60 events/min are found? thx in advance

4 Answers 4

4

Ideally, you'd want to try and process the file only once and store as little of it in memory. In awk, you could do:

awk -v n=60 '
  {
    t = $2 substr($3, 1, 5);
    if (t == last_t) {
      if (++lines > n)
        print
      else
        if (lines == n)
          print saved $0
        else
          saved = saved $0 RS
    } else {
      saved = $0 RS
      lines = 1
      last_t = t
    }
  }' < your-file

A few advantages with that approach:

  • That's stream-processing oriented. Input is processed as soon as it comes and output is issued as soon as possible (as soon as the 60th line has been seen). That makes it possible to post-process process live output (like on a tail -fn +1 log_file).
  • it runs only one invocation of one command (awk), so is going to be as efficient as one can get. The opposite extreme would be to run several commands in a loop. The costliest thing in shell scripting is generally forking and executing commands. Optimising means reducing that as much as possible.
  • we only store at most 60 lines in memory so the memory usage will be bound (provided the size of the lines themselves is bound).
  • awk code can be made very legible and self explanatory. Now, if size's the matter, you can also make it shorter and on one line like

    awk '{t=$2substr($3,1,5);if(t==l){if(++i>n)print;else if(i==n)print s$0;else s=s$0RS}else{s=$0RS;i=1;l=t}}' n=60 file
    
2

This is not the most efficient solution, but you can first count the number of events for every minute, and then grep your file for each of these minutes when the count is >= 60.

sort -k 2,3 your_log_file \
| uniq -c -s 8 -w 16 \
| while read count _ date time _; do
    [ "$count" -ge 60 ] && grep -F " $date ${time%:*}" your_log_file
done

Notes:

  • in the basic example above, I have first sorted your file chronologically
  • the first two lines will give you the number of events per minutes, if this is the only information that interests you.

If your file is full of events, chances are that you'll end up doing numerous greps on it. A better solution would be to read the log file sequentially, and memorize the lines of the last minute. When you reach the next minute, print these lines if their number is greater than 60. See Stéphane's answer for such a solution.

2
  • The events could also be slightly disordered, that is I can have an event at 00:19 but then one at 00.18 and then 00:19 again. Btw I tried your solution (even changing 60 to lower values like 30) but it doesn't print anything.
    – Dad85
    Dec 20, 2016 at 10:53
  • @Dad85 I have changed my answer to add first a chronological sort. I also fixed a mistake of mine. The script should work now.
    – xhienne
    Dec 20, 2016 at 11:08
1

With something like this you can isolate the minutes available:

root@debian:# awk -F" " '{print $2" "$3}' b.txt |cut -f1-2 -d: |uniq
01/01/2010 20:56
02/01/2010 01:39
02/01/2010 01:40
02/01/2010 20:56

You can then assign an array with those values

Revised Code:

readarray -t stamps < <(awk -F" " '{print $2" "$3}' b.txt |cut -f1-2 -d: |uniq)
for stamp in "${stamps[@]}";do
ev=$(grep "$stamp" b.txt |wc -l)
echo "In $stamp found $ev events "
#if [ "$ev" -gt 60 ]; then
# do the stuff
#fi
done

Output:

In 01/01/2010 20:56 found 7 events 
In 02/01/2010 01:39 found 11 events 
In 02/01/2010 01:40 found 4 events 
In 02/01/2010 20:56 found 7 events 
2
  • 2
    this looks horribly inefficient - you should use uniq -c to get the count for each minute instead of running grep | wc -l for each unique minute observed.
    – Alnitak
    Dec 20, 2016 at 14:31
  • 2
    (and FWIW, with 48 hours in the file, and assuming at least one event per minute, that's 2880 times you fork grep and wc, instead of using just one call to uniq -c)
    – Alnitak
    Dec 20, 2016 at 16:05
1
awk '{ print $2 " " $3 }' < input \
| cut -c1-16                      \
| sort                            \
| uniq -c                         \
| awk '{ if ($1 > 60) print $2 }'

i.e. get the date and time fields, strip off the seconds, sort the result (NB: would work better if your dates were in ISO format), find the count of each unique date / time combo, and then print those with a count > 60

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .