The GNU parallel grepping n lines for m regular expressions example states the following:
If the CPU is the limiting factor parallelization should be done on the regexps:
cat regexp.txt | parallel --pipe -L1000 --round-robin grep -f - bigfile
This will start one grep per CPU and read bigfile one time per CPU, but as that is done in parallel, all reads except the first will be cached in RAM
So in this instance GNU parallel
round robins regular expressions from regex.txt
over parallel grep instances with each grep
instance reading bigfile
separately. And as the documentation states above, disk caching probably ensures that bigfile
is read from disk only once.
My question is this - the approach above appears to be seen as better performance-wise than another that involves having GNU parallel
round robin records from bigfile
over parallel grep
instances that each read regexp.txt
, something like
cat bigfile | parallel --pipe -L1000 --round-robin grep -f regexp.txt -
Why would that be? As I see it assuming disk caching in play, bigfile
and regexp.txt
would each be read from disk once in either case. The one major difference that I can think of is that the second approach involves significantly more data being passed through pipes.
-n
option togrep
and get line numbers inbigfile
.grep -f regexp.txt bigfile