grep: Which *patterns* are matched, not which text?

Question

In maintaining our local logcheck rules, it'd be nice to know which ones are no longer being used. I'm trying to find a reasonably efficient way of doing that.

In other words, given a fairly large number of GNU grep extended regexp (grep -E) patterns (around 700), and a lot of syslog output (tens if not hundreds of millions of lines), I'd like to determine which of those ≈700 patterns match none of the syslog lines. Even better would be to find out how many syslog lines each pattern matches.

The obvious way is to run grep -c -E "$pattern" «massive-logfile» ≈700 times, once per pattern. But that seems inefficient, especially if the massive logfile doesn't fit in RAM.

Is there some efficient way to do this instead?

Answer 1

You could use awk whose regexps are very similar to grep -E's:

awk '!patterns_read{patterns[$0]; next}
     {for (p in patterns) if ($0 ~ p) c[p]++}
     END {
       for (p in patterns) printf "'%s' was matched %d times\n", p, c[p]
     }' patterns patterns_read=1 log files

Answer 2

For each pattern:

if ! grep -q "$pattern" /path/to/input; then
    echo "/${pattern}/ not found."
fi

This is AFAIK really the only way to do it, because if you search for multiple patterns at once, you will only know that you've matched at least one of them. You could try, if you have a lot of logical ORs (e. g. /(needle|pin)/, starting by just iterating through each of those permutations, but you will still have to test one expression at a time.

Answer 3

Depending on how complex the REs are, you might be able to use a language like Python (untested):

#! /usr/bin/env python3
import re, sys
res = ["re1", "re2", ... ]  # or read from a file
recs = [re.compile(r) for r in res]
matches = {}
for line in sys.stdin:
    for r in recs:
        if r.match(line):
            matches[r] += 1
for r in matches:
    if matches[r] == 0:
        print(r.pattern)

Of course, if the regexes contain strings that might not be taken literally by Python's more advanced regex support, this won't work. You could probably eliminate a lot of work by feeding the output of grep to this (since then we eliminate lines on which no regex matches).

Answer 4

perl -lne '
   # read in the patterns into a hash
   @ARGV and $h{$_}=s|/|\\/|gr,next;

   # delete pattern if matched, so we wont have to
   # expend efforts on it for the subsequent lines
   while (my($pat) = each %h) {
      delete $h{$pat} if /$h{$pat}/;
   }

   # what remains are those that did not match
   END {
      print "These patterns did not match:";
      print for keys %h;
   }
' patterns_file log_file