I have a directory of files that is structured like this:


There are thousands of sub directories within data/, each with one xml file.

I would like to recursively scan data/ and look for three patterns, then echo the count of the pattern match (ideally) with a label before each. So, for my output, I'd like to see

Pattern 1 = 20
Pattern 2 = 100
Pattern 3 = 432

I've been able to use this to run these one at a time using:

grep -E -r 'Pattern 1' data/ | wc -l
grep -E -r 'Pattern 2' data/ | wc -l
grep -E -r 'Pattern 3' data/ | wc -l 

Now I'm trying to put this all into a shell script with one argument, which would be parent directory of data/. Then output the count to each pattern to either standard output or count.txt.

Your help is appreciated!

  • 3
    have you check -c flag to grep ? – Rahul Patil Jan 15 '13 at 15:58
  • Of course, grep -c makes sense! – karavan Jan 15 '13 at 16:12

You should check grep -c, it does exactly what you describe. If you need more flexibility, you could probably use awk. Try something like this (typed directly in the browser and not tested):

# add tests to check if $1 exists (-d) and/or is non null (-n)

find "${dir}" | awk <<\EOF

   pat1 = 0;
   pat2 = 0;

/pattern/ { pat1 += 1;}
/otherpattern/ {pat2 += 2;}

   print "pattern", pat1;
   print "otherpattern", pat2;

I'll leave it up to you to work out the details of the find command or the awk script.

| improve this answer | |
  • Thanks for this. I know nothing about awk so this is entirely new territory. I guess I was hoping the for loop would work and I wouldn't have to use something like awk. – karavan Jan 15 '13 at 16:39
  • Of course the for loop can work. Try this for pat in 'pat1' 'pat2' 'pat3'; do echo $pat $(grep -E -rc "$pat" $1); done. Depending on the version of grep you're using, you may need to tweak this command a (tiny) bit. I only mentioned awk so that you loop only once over the subdirs (in case there are too many) and you want to control the output or add some calculation. The awk script I wrote is somewhat incomplete but should give you an idea of how it works. Try to read it I bet you'd find out awk isn't as hard as it's made out to be :) – rahmu Jan 15 '13 at 16:40
  • Thanks, I will do my awk homework :) – karavan Jan 15 '13 at 17:04
  • Trying to figure out why the for loop from @rahmu above doesn't work. I did: for pat in 'text1' 'text2' 'text3'; do echo $(grep -E -rc '$pat' $1); done' The result is something like this for every file in the directory (there are thousands): "/data/dir1/file:0". What I am expecting is "text1=10, text2=20, text3=32". Thoughts? – karavan Jan 15 '13 at 18:52

The following script loops over the pattern and prints the total count of matches for each pattern. It searches under the directory whose name is passed as an argument, or under the current directory if you don't pass an argument.

for p in 'Pattern 1' 'Pattern 2' 'Pattern 3; do
  printf '%s = ' "$p"; grep -E -r -e "$p" "${1:-.}/data" | wc -l
| improve this answer | |
  • That's exactly it! Thank you! I would vote up your answer but I don't have enough reputation points :). Can you explain this ${1:-.} a little? I understand that it is the variable from the arguement, but what does :- do? – karavan Jan 16 '13 at 16:58
  • @karavan: This is the syntax for defining a default value. If $1 is empty (if no command line arguments are supplied), then use to value .. man bash and a quick search of :- would explain it best. – rahmu Jan 19 '13 at 15:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.