Add mtime to grep -c output and sort the output by mtime

Question

I have a directory full of logs named in the following style:

info.log00001
info.log00002
info.log00003
...
info.log09999
info.log

My current output (using grep -c)

I need to analyze the frequency a particular error that happens occasionally, so go to that directory and use grep -crw . -e "FooException BarError" | sort -n | less obtaining something like:

./info.log00001: 1
./info.log00002: 0
./info.log00003: 42
...
./info.log09999: 25
./info.log: 0

Then, I can ls -lt to see their modification date and analyze when the error happened the most.

My desired output (with count and date)

Anyway, I'd like find a way to get an output with the count and the date in the same line. That would make my analysis easier. I would like something as:

2015-09-31 10:00 ./info.log00001: 1
2015-09-31 10:15 ./info.log00002: 0
2015-09-31 10:30 ./info.log00003: 42
...
2016-04-01 13:20 ./info.log09999: 25
2015-09-31 13:27 ./info.log: 0

Additional info

Ideally, I'd like to accomplish this with only one command, but first throwing grep's output to a file and then processing that file would make it, too.

Also, I really don't care about the date format or whether the date is at the end or at the beginning of the line. All I want to to is to have the files sorted by date starting with the oldest (which is also the file with the lowest number in its name)

I found a way to accomplish something similar with awk, but in my case it would not work, since it parses the filename from grep's output, and in my case, grep's output has more text that just the path to the file.

I'd really appreciate any feedback on this.

Answer 1

If you have gnu find - and assuming none of your file names contains newlines - you could use find's -printf to output the mtime in the desired format + the file name then run grep to get the count:

find . -type f -printf '%TY-%Tm-%Td %TH:%TM %p: ' -exec grep -cw "whatever" {} \; | sort -k1,1 -k2,2

---

Alternatively, with zsh you could glob and sort by modification time (via glob qualifiers - . selects regular files, Om sorts in descending order by mtime) and then for each file print the mtime using the stat module, the file name and then, again, get the count via grep:

zmodload zsh/stat
for f in ./**/*(.Om)
do
printf '%s %s\t%s %s: ' $(zstat -F '%Y-%b-%d %H:%M' +mtime -- $f) $f
grep -cw "whatever"  $f
done

Answer 2

A bash construct

shopt -s globstar
join -o 1.2,0,2.2 -t$'\034' <(stat -c $'%n\034%y' ** | sort) \
                            <(grep -crw "..." | sort | sed -r $'s/:([^:]*)/\034\\1/') \
                             | tr '\034' '\t'

There's some funky stuff in there with \034 -- that's the octal value for the FS character. It is conceivable that you have existing filenames with that character in the name, you can change that delimiter char if you need to.

The stat command spits out the filename and the modified time for each file recursively.

You are familiar with the grep command. I translate the last colon to the FS char.

I have to use bash's ANSI-C quoting syntax for both stat and sed to get the FS char in there.

The join command joins the output of stat with that of grep, and outputs "date FS filename FS count"

Then I translate FS to a plain tab.

Answer 3

Perl based solution:

perl -lne 'if ($.==1) {
 print localtime($t)." $f: $c\n" if defined $t; 
 $c=0; $f=$ARGV; $t=(stat($f))[7];} $c++ if /$expr/o; 
} BEGIN { $expr=shift @ARGV; push @ARGV,"/etc/hosts"; 
' "search-expression" info.log

Note: untested.

Some standard perl trickery here. -n wraps while (<>) { around your code. When $. is 1, it's a new file. If we have processed a file, print out the summary information - timestamp, filename, count. Now, for the first line only, get the current filename, the file's mtime timestamp, and reset the counter. For each line, increment the counter if it matches the desired regular expression. End the while loop and start some initiation code, which is executed before the previous block. Take the first argument as the regex to match on each line. Now, to the argument list, remove the first argument. Then, append a dummy file to the arg list which will trigger the final print expression. It's a bit of a harmless hack.