I'm running this command on my Mac OS X on 8 cores:

for i in $(cat file1);do grep "$i" file2; done > output.txt

My file2 is 9 million lines so it takes a lot of time. Is it possible to use all the cores to split the job and do everything faster?

  • Check if that version of grep knows the --mmap option, it should speed it up a bit.
    – ott--
    Feb 12 '15 at 14:21

There is other way to make things faster: Use grep -f file1 file2 >output.txt.

You could also use gnu parallel: http://www.gnu.org/software/parallel/parallel_tutorial.html


grep -F -f needles.txt haystack.log is what you want.

-F makes grep use simple pattern matching, if you do not need full regex. This tends to be a HUGE speedup. Depending on what you're searching for, you might not need regex, so might as well get the benefits of faster execution of simpler code.

-f is a file of patterns you're looking for. So instead of doing full N scans for N patterns, you can do it in one pass. This is also a huge speedup on any files that are too big to fit into RAM.

Put the file you're searching through on a fast disk. SSD is not necessarily the answer here, as grep does a scan in a very linear fashion, which is just fine with a normal spinning disk, and especially fast when you have an array of them.

The combination of the two switches, plus the HDD speed can make grep be very fast. If your data is highly repetitive (nicely compressible) you might want to try to store it gzipped, and search through it with zgrep. This helps extra if your storage isn't particularly fast.

If you're responsible for how the data goes into the files, make things small. Do not waste too much space for structure, and take out data that's not carrying any information (think -'s inside of UUID, or spaces and newlines between XML tags), makes logs smaller, which makes both the disk reading time and the parsing time shorter.

Sometimes it's actually faster to do a double parse. This is highly data dependent. For example, if you're looking for a piece of data that's complicated to parse, but it occurs only in small amount of entries easily identifiable by another easy to parse expression, do a first pass for the easy expression. This cuts down the data that you need to run the slow/complicated expression on.

If these don't help, then split -l the files, and run grep with parallel. Remember that the total time of splitting/writing multiple files, and then having multiple processes trying to use the same disk at the same time might be detrimental to the total time the job takes. If you can split the big file into smaller files on multiple disks, then yes, the searches will probably be faster that way.

One common mistake I've seen people do, is try to put the split files on ramdisk. That's you doing caching, badly. The OS will most likely do a better job using that same amount of ram for caching what you're actually working on, not the entire file.

Also, you want to monitor your CPU and disk activity as you're playing with the different scenarios, and learn the difference between 'bottleneck' and 'saturation.'


Based on http://www.gnu.org/software/parallel/man.html#EXAMPLE:-Grepping-n-lines-for-m-regular-expressions

parallel -k parallel --pipepart --block 100M -a file2 -k grep {} :::: file1

Simpler and maybe faster:

parallel -j0 -k grep {} file2 :::: file1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.