Currently I am grepping data from a file containing any of the following:


This list is longer and contaings ~700 different values.

What is the most efficient way of extracting it? I can chop it in parts of 10/20/50/100... Or are there other unix methods? This grep is piped to python for further analysis which goes fast enough.

  • 1
    I don't quite understand what you're asking, can you elaborate? Your source file has a single line with lots of values in it? Or the text you've included is what you're looking for in the source file? Oct 23, 2012 at 9:35

4 Answers 4


Put all those numbers in a file, each on its own line, then use grep with -f or --file= switch:

grep -f filewithcodestosearchfor filetosearchin
  • Hmm the first few runs it seemed faster but that might be human interpretation. Need to run time on it as well. Is it also possible to delete the grepped lines from the file?
    – Jasper
    Oct 23, 2012 at 9:25
  • Is that actually faster or just more convenient to work with?
    – Marco
    Oct 23, 2012 at 9:27
  • 3
    @Marco, it would be faster with --fixed-strings aka -F aka fgrep
    – poige
    Oct 23, 2012 at 9:31
  • @poige The performance increase comes from -F (which is not mentioned in this answer) and not from -f, right?
    – Marco
    Oct 23, 2012 at 9:36
  • @Marco, why I suggested -f is to avoid Jasper's initial intention of using multiple grep calls, each with a subset of the code list. grep, especially GNU grep, is strong in optimization, is better to just let it work instead of spoonfeeding it. Of course, @poige's -F still improves a lot. pastebin.com/P8tTEmie
    – manatwork
    Oct 23, 2012 at 10:06

Have you tried using the --mmap flag?

From man:

   --mmap If possible, use the mmap(2) system call to read input, instead
          of the default read(2) system call.  In some situations, --mmap
          yields better performance.  However, --mmap can cause undefined
          behavior (including core dumps) if an input file shrinks while
          grep is operating, or if an I/O error occurs.

No need to put them in a file, you can do:

grep -F '342163477
168490006' file

as well, which is just as portable/standard (as using grep -Ff, \| in grep BREs is not standard/portable).

Also, what's the real question? Do you want to get the lines that contain those numbers or to know which of those numbers appear in the file? Is the file sorted? Is there one number per line in the file?...


As poige mentions in a comment to another answer, use grep -F/fgrep if you can. This treats any regex characters as literals. You can still get the effect of alternation by combining this with the -f file option, and putting your patterns in file, one pattern to a line.

If you need your patterns to include regex, then try limiting yourself to classic ERE syntax, and use grep -E/egrep. Classic ERE syntax doesn't include backreferences, so no patterns of the form ...(pat)...\1... Potentially this may use a DFA to perform the match, and be more efficient. But that depends on your egrep implementation. More discussion of this is here. Note that some egrep implementations like Gnu's will accept patterns with backreferences; but when matching those patterns the efficiency advantage mentioned here will certainly not be present.

I mention the egrep optimization for theoretical interest, and because it'd be worth testing to see if it helps in your intended use. I'd guess that, as manatwork says, Gnu's grep engine is better at figuring out how to optimize than you are.

  • It's already been said in existent answer, why making dups, just to make bonuses? That's crumby.
    – poige
    Oct 23, 2012 at 11:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.