2

I have a directory full of logs named in the following style:

info.log00001
info.log00002
info.log00003
...
info.log09999
info.log


My current output (using grep -c)

I need to analyze the frequency a particular error that happens occasionally, so go to that directory and use grep -crw . -e "FooException BarError" | sort -n | less obtaining something like:

./info.log00001: 1
./info.log00002: 0
./info.log00003: 42
...
./info.log09999: 25
./info.log: 0

Then, I can ls -lt to see their modification date and analyze when the error happened the most.


My desired output (with count and date)

Anyway, I'd like find a way to get an output with the count and the date in the same line. That would make my analysis easier. I would like something as:

2015-09-31 10:00 ./info.log00001: 1
2015-09-31 10:15 ./info.log00002: 0
2015-09-31 10:30 ./info.log00003: 42
...
2016-04-01 13:20 ./info.log09999: 25
2015-09-31 13:27 ./info.log: 0


Additional info

Ideally, I'd like to accomplish this with only one command, but first throwing grep's output to a file and then processing that file would make it, too.

Also, I really don't care about the date format or whether the date is at the end or at the beginning of the line. All I want to to is to have the files sorted by date starting with the oldest (which is also the file with the lowest number in its name)

I found a way to accomplish something similar with awk, but in my case it would not work, since it parses the filename from grep's output, and in my case, grep's output has more text that just the path to the file.

I'd really appreciate any feedback on this.

  • 2
    Are the date-times inside the logs? Or must you depend on the file's timestamp? If the former, consider installing "histogram", a wrapper around awk. github.com/otheus/histogram If the latter, I could extend its functionality. – Otheus Apr 1 '16 at 16:52
  • Actually, I want to print the timestamp of the last modification of each log. Something similar to what you get when doing ls -lt – Sam Apr 1 '16 at 16:54
  • The answer you referenced uses awk to invoke the "stat" command. You could do that within histogram because its just a wrapper for awk, but does all the counting for you. Anyway, I think awk is the most useful tool here. A perl oneliner would also work very well. The trick is to use ONLY awk or perl – Otheus Apr 1 '16 at 16:58
  • I'd wrote one but it's too difficult from my phone' – Otheus Apr 1 '16 at 17:01
  • @don_crissti that's brilliant. Why not make it an answer? – Otheus Apr 1 '16 at 17:15
5

If you have gnu find - and assuming none of your file names contains newlines - you could use find's -printf to output the mtime in the desired format + the file name then run grep to get the count:

find . -type f -printf '%TY-%Tm-%Td %TH:%TM %p: ' -exec grep -cw "whatever" {} \; | sort -k1,1 -k2,2

Alternatively, with zsh you could glob and sort by modification time (via glob qualifiers - . selects regular files, Om sorts in descending order by mtime) and then for each file print the mtime using the stat module, the file name and then, again, get the count via grep:

zmodload zsh/stat
for f in ./**/*(.Om)
do
printf '%s %s\t%s %s: ' $(zstat -F '%Y-%b-%d %H:%M' +mtime -- $f) $f
grep -cw "whatever"  $f
done
  • 1
    These assume you're trying to search through all regular files. Adjust per your needs, e.g. use find with -type f -name info.log\* or zsh with for f in ./**/info.log*(.Om) to match file names as in your example. – don_crissti Apr 1 '16 at 17:40
  • This worked like a charm. Anyway, I'd appreaciate if you'd explain every part of your command so that this can be usefule for future reference for other people. (For instance, why do you need the empty brackets after grep?) – Sam Apr 1 '16 at 18:35
  • This is a nice solution. It will call grep for each file individually so that might slow it down – glenn jackman Apr 1 '16 at 21:31
  • @Sam - I've added some short explanations and links - it's all in the man pages. As to {} : man find states: The string '{}' is replaced by the current file name being processed. @glenn - thanks, yours's nice too. – don_crissti Apr 2 '16 at 13:21
3

A bash construct

shopt -s globstar
join -o 1.2,0,2.2 -t$'\034' <(stat -c $'%n\034%y' ** | sort) \
                            <(grep -crw "..." | sort | sed -r $'s/:([^:]*)/\034\\1/') \
                             | tr '\034' '\t'

There's some funky stuff in there with \034 -- that's the octal value for the FS character. It is conceivable that you have existing filenames with that character in the name, you can change that delimiter char if you need to.

The stat command spits out the filename and the modified time for each file recursively.

You are familiar with the grep command. I translate the last colon to the FS char.

I have to use bash's ANSI-C quoting syntax for both stat and sed to get the FS char in there.

The join command joins the output of stat with that of grep, and outputs "date FS filename FS count"

Then I translate FS to a plain tab.

  • Hah! I'm glad you answered this one! I thought about join, but as a one liner, it's beyond my ways of thinking. :) – Otheus Apr 1 '16 at 17:45
3

Perl based solution:

perl -lne 'if ($.==1) {
 print localtime($t)." $f: $c\n" if defined $t; 
 $c=0; $f=$ARGV; $t=(stat($f))[7];} $c++ if /$expr/o; 
} BEGIN { $expr=shift @ARGV; push @ARGV,"/etc/hosts"; 
' "search-expression" info.log

Note: untested.

Some standard perl trickery here. -n wraps while (<>) { around your code. When $. is 1, it's a new file. If we have processed a file, print out the summary information - timestamp, filename, count. Now, for the first line only, get the current filename, the file's mtime timestamp, and reset the counter. For each line, increment the counter if it matches the desired regular expression. End the while loop and start some initiation code, which is executed before the previous block. Take the first argument as the regex to match on each line. Now, to the argument list, remove the first argument. Then, append a dummy file to the arg list which will trigger the final print expression. It's a bit of a harmless hack.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.