3

In *nix terminal (Bash) os there any other faster way to find a pattern (or excluding them - egrep -v option) than using this:

grep -f {patternfile} {source}

Using this option is painfully slow for larger files.

Improve this question
4
  • @IgnacioVazquez-Abrams not quite, you can match them in O(N + M), N the file length, M the sum of all pattern lengths
    – hroptatyr
    Aug 16, 2013 at 9:33
  • 1
    Try LC_ALL=C grep -f patternfile source if applicable
    – Stéphane Chazelas
    Aug 16, 2013 at 11:33
  • What's a bash tool?
    – Stéphane Chazelas
    Aug 16, 2013 at 11:34
  • 2
    @StephaneChazelas - a tool for bashing stuff?
    – slm ♦
    Aug 16, 2013 at 18:46

2 Answers 2

Reset to default
3

If your patterns are fixed strings, have a look at agrep (download) (or other matchers that are based on the Wu-Manber algorithm). It will match all patterns in parallel.

Wu-Manber is generally good for up to a few hundred thousand patterns (maybe a million at a push).

For regular expressions I think there's no generally fast approach.

Improve this answer
3
  • Unfortunately agrep does not support reading patterns from file and for my arguments list is too long. "/usr/bin/agrep: Argument list too long"
    – tiny
    Aug 16, 2013 at 10:36
  • Oh, didn't know that, I couldn't use agrep either. I've implemented the Wu-Manber myself because I needed a twist though: matching ~200k - 900k patterns against 10+ million files, only report matches ... don't suppose that's of any use?
    – hroptatyr
    Aug 16, 2013 at 10:59
  • 1
    Um, agrep (from ftp.cs.arizona.edu/agrep) does support patterns from a file, btw, it's -f
    – hroptatyr
    Aug 16, 2013 at 11:01
0

You don't specify what type of file you're searching through but another tool that I've seen used for searching which might give you better performance is ack.

It doesn't however provide the ability to search using a pattern file. However you can use this approach to coax ack into searching for multiple patterns like so:

  $ grep -Ffx <(ack -l "pattern1" file) <(ack -l "pattern2" file)

This could easily be put into a Bash script making it more general purpose.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .