In maintaining our local logcheck rules, it'd be nice to know which ones are no longer being used. I'm trying to find a reasonably efficient way of doing that.

In other words, given a fairly large number of GNU grep extended regexp (grep -E) patterns (around 700), and a lot of syslog output (tens if not hundreds of millions of lines), I'd like to determine which of those ≈700 patterns match none of the syslog lines. Even better would be to find out how many syslog lines each pattern matches.

The obvious way is to run grep -c -E "$pattern" «massive-logfile» ≈700 times, once per pattern. But that seems inefficient, especially if the massive logfile doesn't fit in RAM.

Is there some efficient way to do this instead?

  • you do not need to match all pattern on all lines, just find first 100 occurences for each pattern and exit, this way you'll remember only pattern matched less than 100 time (at the price of going to all log).
    – Archemar
    Aug 10, 2017 at 16:09

4 Answers 4


You could use awk whose regexps are very similar to grep -E's:

awk '!patterns_read{patterns[$0]; next}
     {for (p in patterns) if ($0 ~ p) c[p]++}
     END {
       for (p in patterns) printf "'%s' was matched %d times\n", p, c[p]
     }' patterns patterns_read=1 log files
  • GNU awk seems to be pretty close, including things like /[[:digit:]]/. The docs make it look like it's the same. Any idea how they're different?
    – derobert
    Aug 10, 2017 at 16:37
  • 1
    @derobert, per POSIX, they're meant to be the same but that's a lie. There are subtle differences but you'll find as many differences between two awk implementations than between two egreps or one awk and one egrep. The notable one is p='[\]' awk '$0 ~ ENVIRON["p"]' which fails to match backslash in most implementations while egrep '[\]' works in most egrep implementations. Aug 10, 2017 at 16:44
  • Would GNU awk and GNU grep with -E support the same regex?
    – muru
    Aug 10, 2017 at 16:49
  • 2
    @muru, anyway the list of things supported by GNU grep varies from one version to the next, so you'd be lucky if one version of gawk supported the exact same operators as one version of GNU grep. In any case, see the example above of [\] that works in GNU grep -E, but not gawk. Aug 10, 2017 at 16:55

For each pattern:

if ! grep -q "$pattern" /path/to/input; then
    echo "/${pattern}/ not found."

This is AFAIK really the only way to do it, because if you search for multiple patterns at once, you will only know that you've matched at least one of them. You could try, if you have a lot of logical ORs (e. g. /(needle|pin)/, starting by just iterating through each of those permutations, but you will still have to test one expression at a time.


Depending on how complex the REs are, you might be able to use a language like Python (untested):

#! /usr/bin/env python3
import re, sys
res = ["re1", "re2", ... ]  # or read from a file
recs = [re.compile(r) for r in res]
matches = {}
for line in sys.stdin:
    for r in recs:
        if r.match(line):
            matches[r] += 1
for r in matches:
    if matches[r] == 0:

Of course, if the regexes contain strings that might not be taken literally by Python's more advanced regex support, this won't work. You could probably eliminate a lot of work by feeding the output of grep to this (since then we eliminate lines on which no regex matches).

  • Some of them are unfortunately complicated. And even more unfortunately most of the lines should match a regexp—anything that doesn't match gets sent to the sysadmin in an email, so almost everything does.
    – derobert
    Aug 10, 2017 at 16:23
  • @derobert I suppose "complex" is the wrong word here - I meant things like \d which wouldn't be in GNU grep+ERE but would match digits in Python's re, but, yes, Stephane's implementation in awk would be easier than using Python
    – muru
    Aug 10, 2017 at 16:34
  • Ah. Things I have after backslash are: ( ) [ ] . s S w * \ / # ? + ^, with \s, \S, and \w meaning something to grep. So I'm guessing it'd be fine...
    – derobert
    Aug 10, 2017 at 16:42
perl -lne '
   # read in the patterns into a hash
   @ARGV and $h{$_}=s|/|\\/|gr,next;

   # delete pattern if matched, so we wont have to
   # expend efforts on it for the subsequent lines
   while (my($pat) = each %h) {
      delete $h{$pat} if /$h{$pat}/;

   # what remains are those that did not match
   END {
      print "These patterns did not match:";
      print for keys %h;
' patterns_file log_file

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .