3

In *nix terminal (Bash) os there any other faster way to find a pattern (or excluding them - egrep -v option) than using this:

grep -f {patternfile} {source}

Using this option is painfully slow for larger files.

4
  • @IgnacioVazquez-Abrams not quite, you can match them in O(N + M), N the file length, M the sum of all pattern lengths
    – hroptatyr
    Commented Aug 16, 2013 at 9:33
  • 1
    Try LC_ALL=C grep -f patternfile source if applicable Commented Aug 16, 2013 at 11:33
  • What's a bash tool? Commented Aug 16, 2013 at 11:34
  • 2
    @StephaneChazelas - a tool for bashing stuff?
    – slm
    Commented Aug 16, 2013 at 18:46

2 Answers 2

3

If your patterns are fixed strings, have a look at agrep (download) (or other matchers that are based on the Wu-Manber algorithm). It will match all patterns in parallel.

Wu-Manber is generally good for up to a few hundred thousand patterns (maybe a million at a push).

For regular expressions I think there's no generally fast approach.

3
  • Unfortunately agrep does not support reading patterns from file and for my arguments list is too long. "/usr/bin/agrep: Argument list too long"
    – tiny
    Commented Aug 16, 2013 at 10:36
  • Oh, didn't know that, I couldn't use agrep either. I've implemented the Wu-Manber myself because I needed a twist though: matching ~200k - 900k patterns against 10+ million files, only report matches ... don't suppose that's of any use?
    – hroptatyr
    Commented Aug 16, 2013 at 10:59
  • 1
    Um, agrep (from ftp.cs.arizona.edu/agrep) does support patterns from a file, btw, it's -f
    – hroptatyr
    Commented Aug 16, 2013 at 11:01
0

You don't specify what type of file you're searching through but another tool that I've seen used for searching which might give you better performance is ack.

It doesn't however provide the ability to search using a pattern file. However you can use this approach to coax ack into searching for multiple patterns like so:

  $ grep -Ffx <(ack -l "pattern1" file) <(ack -l "pattern2" file)

This could easily be put into a Bash script making it more general purpose.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .