In maintaining our local logcheck rules, it'd be nice to know which ones are no longer being used. I'm trying to find a reasonably efficient way of doing that.
In other words, given a fairly large number of GNU grep extended regexp (
grep -E) patterns (around 700), and a lot of syslog output (tens if not hundreds of millions of lines), I'd like to determine which of those ≈700 patterns match none of the syslog lines. Even better would be to find out how many syslog lines each pattern matches.
The obvious way is to run
grep -c -E "$pattern" «massive-logfile» ≈700 times, once per pattern. But that seems inefficient, especially if the massive logfile doesn't fit in RAM.
Is there some efficient way to do this instead?