In *nix terminal (Bash) os there any other faster way to find a pattern (or excluding them - egrep -v option) than using this:

grep -f {patternfile} {source}

Using this option is painfully slow for larger files.

  • @IgnacioVazquez-Abrams not quite, you can match them in O(N + M), N the file length, M the sum of all pattern lengths – hroptatyr Aug 16 '13 at 9:33
  • 1
    Try LC_ALL=C grep -f patternfile source if applicable – Stéphane Chazelas Aug 16 '13 at 11:33
  • What's a bash tool? – Stéphane Chazelas Aug 16 '13 at 11:34
  • 1
    @StephaneChazelas - a tool for bashing stuff? – slm Aug 16 '13 at 18:46

If your patterns are fixed strings, have a look at agrep (download) (or other matchers that are based on the Wu-Manber algorithm). It will match all patterns in parallel.

Wu-Manber is generally good for up to a few hundred thousand patterns (maybe a million at a push).

For regular expressions I think there's no generally fast approach.

  • Unfortunately agrep does not support reading patterns from file and for my arguments list is too long. "/usr/bin/agrep: Argument list too long" – tiny Aug 16 '13 at 10:36
  • Oh, didn't know that, I couldn't use agrep either. I've implemented the Wu-Manber myself because I needed a twist though: matching ~200k - 900k patterns against 10+ million files, only report matches ... don't suppose that's of any use? – hroptatyr Aug 16 '13 at 10:59
  • 1
    Um, agrep (from ftp.cs.arizona.edu/agrep) does support patterns from a file, btw, it's -f – hroptatyr Aug 16 '13 at 11:01

You don't specify what type of file you're searching through but another tool that I've seen used for searching which might give you better performance is ack.

It doesn't however provide the ability to search using a pattern file. However you can use this approach to coax ack into searching for multiple patterns like so:

  $ grep -Ffx <(ack -l "pattern1" file) <(ack -l "pattern2" file)

This could easily be put into a Bash script making it more general purpose.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.