Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have a directory of files that is structured like this:

data/directory1/file1.xml
data/directory2/file2.xml
data/directory3/file3.xml
...

There are thousands of sub directories within data/, each with one xml file.

I would like to recursively scan data/ and look for three patterns, then echo the count of the pattern match (ideally) with a label before each. So, for my output, I'd like to see

Pattern 1 = 20
Pattern 2 = 100
Pattern 3 = 432

I've been able to use this to run these one at a time using:

grep -E -r 'Pattern 1' data/ | wc -l
grep -E -r 'Pattern 2' data/ | wc -l
grep -E -r 'Pattern 3' data/ | wc -l 

Now I'm trying to put this all into a shell script with one argument, which would be parent directory of data/. Then output the count to each pattern to either standard output or count.txt.

Your help is appreciated!

share|improve this question
2  
have you check -c flag to grep ? –  Rahul Patil Jan 15 '13 at 15:58
    
Of course, grep -c makes sense! –  karavan Jan 15 '13 at 16:12
add comment

2 Answers

You should check grep -c, it does exactly what you describe. If you need more flexibility, you could probably use awk. Try something like this (typed directly in the browser and not tested):

dir=$1
# add tests to check if $1 exists (-d) and/or is non null (-n)

find "${dir}" | awk <<\EOF

BEGIN {
   pat1 = 0;
   pat2 = 0;
   ...
}

/pattern/ { pat1 += 1;}
/otherpattern/ {pat2 += 2;}
...

END {
   print "pattern", pat1;
   print "otherpattern", pat2;
   ...
}
EOF

I'll leave it up to you to work out the details of the find command or the awk script.

share|improve this answer
    
Thanks for this. I know nothing about awk so this is entirely new territory. I guess I was hoping the for loop would work and I wouldn't have to use something like awk. –  karavan Jan 15 '13 at 16:39
    
Of course the for loop can work. Try this for pat in 'pat1' 'pat2' 'pat3'; do echo $pat $(grep -E -rc "$pat" $1); done. Depending on the version of grep you're using, you may need to tweak this command a (tiny) bit. I only mentioned awk so that you loop only once over the subdirs (in case there are too many) and you want to control the output or add some calculation. The awk script I wrote is somewhat incomplete but should give you an idea of how it works. Try to read it I bet you'd find out awk isn't as hard as it's made out to be :) –  rahmu Jan 15 '13 at 16:40
    
Thanks, I will do my awk homework :) –  karavan Jan 15 '13 at 17:04
    
Trying to figure out why the for loop from @rahmu above doesn't work. I did: for pat in 'text1' 'text2' 'text3'; do echo $(grep -E -rc '$pat' $1); done' The result is something like this for every file in the directory (there are thousands): "/data/dir1/file:0". What I am expecting is "text1=10, text2=20, text3=32". Thoughts? –  karavan Jan 15 '13 at 18:52
add comment

The following script loops over the pattern and prints the total count of matches for each pattern. It searches under the directory whose name is passed as an argument, or under the current directory if you don't pass an argument.

#!/bin/sh
for p in 'Pattern 1' 'Pattern 2' 'Pattern 3; do
  printf '%s = ' "$p"; grep -E -r -e "$p" "${1:-.}/data" | wc -l
done
share|improve this answer
    
That's exactly it! Thank you! I would vote up your answer but I don't have enough reputation points :). Can you explain this ${1:-.} a little? I understand that it is the variable from the arguement, but what does :- do? –  karavan Jan 16 '13 at 16:58
    
@karavan: This is the syntax for defining a default value. If $1 is empty (if no command line arguments are supplied), then use to value .. man bash and a quick search of :- would explain it best. –  rahmu Jan 19 '13 at 15:08
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.