Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have a (hopefully) interesting problem that I could use some advice on. I have a system that's essentially used for storing logs. It has a directory structure like:

YYYY/MM/DD/hostname/

There are a number of hostnames, and under each one are a bunch of gzipped hourly logs (access, error, etc).

What I'm interested in is the total count of a given string in the access logs broken down by day and hostname. What's the best way to do this? Is this possible with a find and grep combination, or is it too complicated for that and instead need a script?

Thanks!

share|improve this question

2 Answers 2

up vote 5 down vote accepted
for d in */*/*/*; do
  printf '%s: ' "$d"
  zcat -- "$d/"*.gz | grep -Fc STRING
done

would count the number of lines that contain STRING.

Replace grep -Fc STRING with grep -Fo STRING | wc -l (assuming GNU grep) to have the number of occurrences.

Replace zcat with gzip -dc if your zcat doesn't support .gz files.

With zsh and GNU grep, you can shorten it to:

for d (*/*/*/*) zcat $d/*.gz | grep -FcH --label=$d STRING
share|improve this answer

If you want to do this without a standard log analyzer, you can do it in a "single line" of shell from the directory containing your YYYY dirs:

for d in */*/*; do 
  echo -n "$d : "
  find $d -name *.log | xargs grep -c STRING
done
share|improve this answer
1  
Do you need to incorporate something like zcat because the logs are gzipped? –  drs Aug 12 at 19:51
1  
@drs if gzipped you can use zgrep instead of grep. –  Jordan Samuels Aug 12 at 19:52

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.