Getting the combined count of all occurrences of a string under multiple directories?

Question

I have a (hopefully) interesting problem that I could use some advice on. I have a system that's essentially used for storing logs. It has a directory structure like:

YYYY/MM/DD/hostname/

There are a number of hostnames, and under each one are a bunch of gzipped hourly logs (access, error, etc).

What I'm interested in is the total count of a given string in the access logs broken down by day and hostname. What's the best way to do this? Is this possible with a find and grep combination, or is it too complicated for that and instead need a script?

Answer 1

for d in */*/*/*; do
  printf '%s: ' "$d"
  zcat -- "$d/"*.gz | grep -Fc STRING
done

would count the number of lines that contain STRING.

Replace grep -Fc STRING with grep -Fo STRING | wc -l (assuming GNU grep) to have the number of occurrences.

Replace zcat with gzip -dc if your zcat doesn't support .gz files.

With zsh and GNU grep, you can shorten it to:

for d (*/*/*/*) zcat $d/*.gz | grep -FcH --label=$d STRING

Answer 2

If you want to do this without a standard log analyzer, you can do it in a "single line" of shell from the directory containing your YYYY dirs:

for d in */*/*; do 
  echo -n "$d : "
  find $d -name *.log | xargs grep -c STRING
done