In maintaining our local logcheck rules, it'd be nice to know which ones are no longer being used. I'm trying to find a reasonably efficient way of doing that.

In other words, given a fairly large number of GNU grep extended regexp (grep -E) patterns (around 700), and a lot of syslog output (tens if not hundreds of millions of lines), I'd like to determine which of those ≈700 patterns match none of the syslog lines. Even better would be to find out how many syslog lines each pattern matches.

The obvious way is to run grep -c -E "$pattern" «massive-logfile» ≈700 times, once per pattern. But that seems inefficient, especially if the massive logfile doesn't fit in RAM.

Is there some efficient way to do this instead?

  • you do not need to match all pattern on all lines, just find first 100 occurences for each pattern and exit, this way you'll remember only pattern matched less than 100 time (at the price of going to all log). – Archemar Aug 10 '17 at 16:09

You could use awk whose regexps are very similar to grep -E's:

awk '!patterns_read{patterns[$0]; next}
     {for (p in patterns) if ($0 ~ p) c[p]++}
     END {
       for (p in patterns) printf "'%s' was matched %d times\n", p, c[p]
     }' patterns patterns_read=1 log files
  • GNU awk seems to be pretty close, including things like /[[:digit:]]/. The docs make it look like it's the same. Any idea how they're different? – derobert Aug 10 '17 at 16:37
  • 1
    @derobert, per POSIX, they're meant to be the same but that's a lie. There are subtle differences but you'll find as many differences between two awk implementations than between two egreps or one awk and one egrep. The notable one is p='[\]' awk '$0 ~ ENVIRON["p"]' which fails to match backslash in most implementations while egrep '[\]' works in most egrep implementations. – Stéphane Chazelas Aug 10 '17 at 16:44
  • Would GNU awk and GNU grep with -E support the same regex? – muru Aug 10 '17 at 16:49
  • 2
    @muru, anyway the list of things supported by GNU grep varies from one version to the next, so you'd be lucky if one version of gawk supported the exact same operators as one version of GNU grep. In any case, see the example above of [\] that works in GNU grep -E, but not gawk. – Stéphane Chazelas Aug 10 '17 at 16:55

For each pattern:

if ! grep -q "$pattern" /path/to/input; then
    echo "/${pattern}/ not found."

This is AFAIK really the only way to do it, because if you search for multiple patterns at once, you will only know that you've matched at least one of them. You could try, if you have a lot of logical ORs (e. g. /(needle|pin)/, starting by just iterating through each of those permutations, but you will still have to test one expression at a time.


Depending on how complex the REs are, you might be able to use a language like Python (untested):

#! /usr/bin/env python3
import re, sys
res = ["re1", "re2", ... ]  # or read from a file
recs = [re.compile(r) for r in res]
matches = {}
for line in sys.stdin:
    for r in recs:
        if r.match(line):
            matches[r] += 1
for r in matches:
    if matches[r] == 0:

Of course, if the regexes contain strings that might not be taken literally by Python's more advanced regex support, this won't work. You could probably eliminate a lot of work by feeding the output of grep to this (since then we eliminate lines on which no regex matches).

  • Some of them are unfortunately complicated. And even more unfortunately most of the lines should match a regexp—anything that doesn't match gets sent to the sysadmin in an email, so almost everything does. – derobert Aug 10 '17 at 16:23
  • @derobert I suppose "complex" is the wrong word here - I meant things like \d which wouldn't be in GNU grep+ERE but would match digits in Python's re, but, yes, Stephane's implementation in awk would be easier than using Python – muru Aug 10 '17 at 16:34
  • Ah. Things I have after backslash are: ( ) [ ] . s S w * \ / # ? + ^, with \s, \S, and \w meaning something to grep. So I'm guessing it'd be fine... – derobert Aug 10 '17 at 16:42
perl -lne '
   # read in the patterns into a hash
   @ARGV and $h{$_}=s|/|\\/|gr,next;

   # delete pattern if matched, so we wont have to
   # expend efforts on it for the subsequent lines
   while (my($pat) = each %h) {
      delete $h{$pat} if /$h{$pat}/;

   # what remains are those that did not match
   END {
      print "These patterns did not match:";
      print for keys %h;
' patterns_file log_file

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.