4

I have a directory of files that is structured like this:

data/directory1/file1.xml
data/directory2/file2.xml
data/directory3/file3.xml
...

There are thousands of sub directories within data/, each with one xml file.

I would like to recursively scan data/ and look for three patterns, then echo the count of the pattern match (ideally) with a label before each. So, for my output, I'd like to see

Pattern 1 = 20
Pattern 2 = 100
Pattern 3 = 432

I've been able to use this to run these one at a time using:

grep -E -r 'Pattern 1' data/ | wc -l
grep -E -r 'Pattern 2' data/ | wc -l
grep -E -r 'Pattern 3' data/ | wc -l 

Now I'm trying to put this all into a shell script with one argument, which would be parent directory of data/. Then output the count to each pattern to either standard output or count.txt.

Your help is appreciated!

2
  • 3
    have you check -c flag to grep ? Commented Jan 15, 2013 at 15:58
  • Of course, grep -c makes sense!
    – karavan
    Commented Jan 15, 2013 at 16:12

2 Answers 2

4

You should check grep -c, it does exactly what you describe. If you need more flexibility, you could probably use awk. Try something like this (typed directly in the browser and not tested):

dir=$1
# add tests to check if $1 exists (-d) and/or is non null (-n)

find "${dir}" | awk <<\EOF

BEGIN {
   pat1 = 0;
   pat2 = 0;
   ...
}

/pattern/ { pat1 += 1;}
/otherpattern/ {pat2 += 2;}
...

END {
   print "pattern", pat1;
   print "otherpattern", pat2;
   ...
}
EOF

I'll leave it up to you to work out the details of the find command or the awk script.

4
  • Thanks for this. I know nothing about awk so this is entirely new territory. I guess I was hoping the for loop would work and I wouldn't have to use something like awk.
    – karavan
    Commented Jan 15, 2013 at 16:39
  • Of course the for loop can work. Try this for pat in 'pat1' 'pat2' 'pat3'; do echo $pat $(grep -E -rc "$pat" $1); done. Depending on the version of grep you're using, you may need to tweak this command a (tiny) bit. I only mentioned awk so that you loop only once over the subdirs (in case there are too many) and you want to control the output or add some calculation. The awk script I wrote is somewhat incomplete but should give you an idea of how it works. Try to read it I bet you'd find out awk isn't as hard as it's made out to be :)
    – rahmu
    Commented Jan 15, 2013 at 16:40
  • Thanks, I will do my awk homework :)
    – karavan
    Commented Jan 15, 2013 at 17:04
  • Trying to figure out why the for loop from @rahmu above doesn't work. I did: for pat in 'text1' 'text2' 'text3'; do echo $(grep -E -rc '$pat' $1); done' The result is something like this for every file in the directory (there are thousands): "/data/dir1/file:0". What I am expecting is "text1=10, text2=20, text3=32". Thoughts?
    – karavan
    Commented Jan 15, 2013 at 18:52
3

The following script loops over the pattern and prints the total count of matches for each pattern. It searches under the directory whose name is passed as an argument, or under the current directory if you don't pass an argument.

#!/bin/sh
for p in 'Pattern 1' 'Pattern 2' 'Pattern 3; do
  printf '%s = ' "$p"; grep -E -r -e "$p" "${1:-.}/data" | wc -l
done
2
  • That's exactly it! Thank you! I would vote up your answer but I don't have enough reputation points :). Can you explain this ${1:-.} a little? I understand that it is the variable from the arguement, but what does :- do?
    – karavan
    Commented Jan 16, 2013 at 16:58
  • @karavan: This is the syntax for defining a default value. If $1 is empty (if no command line arguments are supplied), then use to value .. man bash and a quick search of :- would explain it best.
    – rahmu
    Commented Jan 19, 2013 at 15:08

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .