Suppose I want to find all the matches in compressed text file:
$ gzcat file.txt.gz | pv --rate -i 5 | grep some-pattern
pv --rate used here for measuring pipe throughput. On my machine it's about 420Mb/s (after decompression).
Now I'm trying to do parallel grep using GNU parallel.
$ gzcat documents.json.gz | pv --rate -i 5 | parallel --pipe -j4 --round-robin grep some-pattern
Now throughput is dropped to a ~260Mb/s. And what is more intresting
parallel process itself is using a lot of CPU. More than
grep processes (but less than
EDIT 1: I've tried different block sizes (
--block), as well as different values for
-L options. Nothing helps me at this point.
What am I doing wrong?