Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

In *nix terminal (Bash) os there any other faster way to find a pattern (or excluding them - egrep -v option) than using this:

grep -f {patternfile} {source}

Using this option is painfully slow for larger files.

share|improve this question
    
@IgnacioVazquez-Abrams not quite, you can match them in O(N + M), N the file length, M the sum of all pattern lengths –  hroptatyr Aug 16 '13 at 9:33
1  
Try LC_ALL=C grep -f patternfile source if applicable –  Stéphane Chazelas Aug 16 '13 at 11:33
    
What's a bash tool? –  Stéphane Chazelas Aug 16 '13 at 11:34
1  
@StephaneChazelas - a tool for bashing stuff? –  slm Aug 16 '13 at 18:46

2 Answers 2

up vote 1 down vote accepted

If your patterns are fixed strings, have a look at agrep (download) (or other matchers that are based on the Wu-Manber algorithm). It will match all patterns in parallel.

Wu-Manber is generally good for up to a few hundred thousand patterns (maybe a million at a push).

For regular expressions I think there's no generally fast approach.

share|improve this answer
    
Unfortunately agrep does not support reading patterns from file and for my arguments list is too long. "/usr/bin/agrep: Argument list too long" –  tiny Aug 16 '13 at 10:36
    
Oh, didn't know that, I couldn't use agrep either. I've implemented the Wu-Manber myself because I needed a twist though: matching ~200k - 900k patterns against 10+ million files, only report matches ... don't suppose that's of any use? –  hroptatyr Aug 16 '13 at 10:59
    
Um, agrep (from ftp.cs.arizona.edu/agrep) does support patterns from a file, btw, it's -f –  hroptatyr Aug 16 '13 at 11:01
    
Thanks using agrep was quickest solution! –  tiny Aug 21 '13 at 8:40

You don't specify what type of file you're searching through but another tool that I've seen used for searching which might give you better performance is ack.

It doesn't however provide the ability to search using a pattern file. However you can use this approach to coax ack into searching for multiple patterns like so:

  $ grep -Ffx <(ack -l "pattern1" file) <(ack -l "pattern2" file)

This could easily be put into a Bash script making it more general purpose.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.