3

I have a (hopefully) interesting problem that I could use some advice on. I have a system that's essentially used for storing logs. It has a directory structure like:

YYYY/MM/DD/hostname/

There are a number of hostnames, and under each one are a bunch of gzipped hourly logs (access, error, etc).

What I'm interested in is the total count of a given string in the access logs broken down by day and hostname. What's the best way to do this? Is this possible with a find and grep combination, or is it too complicated for that and instead need a script?

2 Answers 2

6
for d in */*/*/*; do
  printf '%s: ' "$d"
  zcat -- "$d/"*.gz | grep -Fc STRING
done

would count the number of lines that contain STRING.

Replace grep -Fc STRING with grep -Fo STRING | wc -l (assuming GNU grep) to have the number of occurrences.

Replace zcat with gzip -dc if your zcat doesn't support .gz files.

With zsh and GNU grep, you can shorten it to:

for d (*/*/*/*) zcat $d/*.gz | grep -FcH --label=$d STRING
3

If you want to do this without a standard log analyzer, you can do it in a "single line" of shell from the directory containing your YYYY dirs:

for d in */*/*; do 
  echo -n "$d : "
  find $d -name *.log | xargs grep -c STRING
done
2
  • 1
    Do you need to incorporate something like zcat because the logs are gzipped?
    – drs
    Aug 12, 2014 at 19:51
  • 1
    @drs if gzipped you can use zgrep instead of grep. Aug 12, 2014 at 19:52

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .