Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

Currently I am grepping data from a file containing any of the following:

342163477\|405760044\|149007683\|322391022\|77409125\|195978682\|358463993\|397650460\|171780277\|336063797\|397650502\|357636118\|168490006...............

This list is longer and contaings ~700 different values.

What is the most efficient way of extracting it? I can chop it in parts of 10/20/50/100... Or are there other unix methods? This grep is piped to python for further analysis which goes fast enough.

share|improve this question
1  
I don't quite understand what you're asking, can you elaborate? Your source file has a single line with lots of values in it? Or the text you've included is what you're looking for in the source file? –  EightBitTony Oct 23 '12 at 9:35
add comment

4 Answers

Put all those numbers in a file, each on its own line, then use grep with -f or --file= switch:

grep -f filewithcodestosearchfor filetosearchin
share|improve this answer
    
Hmm the first few runs it seemed faster but that might be human interpretation. Need to run time on it as well. Is it also possible to delete the grepped lines from the file? –  Jasper Oct 23 '12 at 9:25
    
Is that actually faster or just more convenient to work with? –  Marco Oct 23 '12 at 9:27
3  
@Marco, it would be faster with --fixed-strings aka -F aka fgrep –  poige Oct 23 '12 at 9:31
    
@poige The performance increase comes from -F (which is not mentioned in this answer) and not from -f, right? –  Marco Oct 23 '12 at 9:36
    
@Marco, why I suggested -f is to avoid Jasper's initial intention of using multiple grep calls, each with a subset of the code list. grep, especially GNU grep, is strong in optimization, is better to just let it work instead of spoonfeeding it. Of course, @poige's -F still improves a lot. pastebin.com/P8tTEmie –  manatwork Oct 23 '12 at 10:06
show 4 more comments

Have you tried using the --mmap flag?

From man:

   --mmap If possible, use the mmap(2) system call to read input, instead
          of the default read(2) system call.  In some situations, --mmap
          yields better performance.  However, --mmap can cause undefined
          behavior (including core dumps) if an input file shrinks while
          grep is operating, or if an I/O error occurs.
share|improve this answer
add comment

No need to put them in a file, you can do:

grep -F '342163477
405760044
149007683
322391022
77409125
195978682
358463993
397650460
171780277
336063797
397650502
357636118
168490006' file

as well, which is just as portable/standard (as using grep -Ff, \| in grep BREs is not standard/portable).

Also, what's the real question? Do you want to get the lines that contain those numbers or to know which of those numbers appear in the file? Is the file sorted? Is there one number per line in the file?...

share|improve this answer
add comment

As poige mentions in a comment to another answer, use grep -F/fgrep if you can. This treats any regex characters as literals. You can still get the effect of alternation by combining this with the -f file option, and putting your patterns in file, one pattern to a line.

If you need your patterns to include regex, then try limiting yourself to classic ERE syntax, and use grep -E/egrep. Classic ERE syntax doesn't include backreferences, so no patterns of the form ...(pat)...\1... Potentially this may use a DFA to perform the match, and be more efficient. But that depends on your egrep implementation. More discussion of this is here. Note that some egrep implementations like Gnu's will accept patterns with backreferences; but when matching those patterns the efficiency advantage mentioned here will certainly not be present.

I mention the egrep optimization for theoretical interest, and because it'd be worth testing to see if it helps in your intended use. I'd guess that, as manatwork says, Gnu's grep engine is better at figuring out how to optimize than you are.

share|improve this answer
    
It's already been said in existent answer, why making dups, just to make bonuses? That's crumby. –  poige Oct 23 '12 at 11:27
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.