8

Suppose I want to find all the matches in compressed text file:

$ gzcat file.txt.gz | pv --rate -i 5 | grep some-pattern

pv --rate used here for measuring pipe throughput. On my machine it's about 420Mb/s (after decompression).

Now I'm trying to do parallel grep using GNU parallel.

$ gzcat documents.json.gz | pv --rate -i 5 | parallel --pipe -j4 --round-robin grep some-pattern

Now throughput is dropped to a ~260Mb/s. And what is more intresting parallel process itself is using a lot of CPU. More than grep processes (but less than gzcat).

EDIT 1: I've tried different block sizes (--block), as well as different values for -N/-L options. Nothing helps me at this point.

What am I doing wrong?

9

I am really surprised you get 270 MB/s using GNU Parallel's --pipe. My tests usually max out at around 100 MB/s.

Your bottleneck is most likely in GNU Parallel: --pipe is not very efficient. --pipepart, however, is: Here I can get in the order of 1 GB/s per CPU core.

Unfortunately there are a few limitations on using --pipepart:

  • The file must be seekable (i.e. no pipe)
  • You must be able to find the start of a record with --recstart/--recend (i.e. no compressed file)
  • The line number is unknown (so you cannot have a 4-line record).

Example:

parallel --pipepart -a bigfile --block 100M grep somepattern
|improve this answer|||||
  • 1
    Thanks. Is there any reason why --pipe is inefficient? I mean is it some sort of fundamental problem or more of implementation specific. – Denis Bazhenov Feb 4 '15 at 7:53
  • 2
    Yes: GNU Parallel is written in perl, and with --pipe every single byte has to go through the single process, which has to do a bit of processing on each byte. With --pipepart most bytes are never seen by the central process: They are processed by spawned jobs. As it is fairly few lines that is the bottleneck in --pipe I would welcome a C/C++ coder who would rewrite the part which would then be run for people who have C-compiler in their path. – Ole Tange Feb 4 '15 at 15:18
2

grep is very effective - there is no sense in running it parallel. In your command only decompression need more cpu, but this can't be paralleled.

Splitting input by parallel need more cpu than get matching lines by grep.

Situation change if you wish to use instead of grep something what need much more cpu for each line - then parallel would have more sense.

If you wish speed up this operation - look where are bottleneck - probably it's decompression (then helps using other decompression tool or better cpu) or - reading from disk (then help using other decompression tool or better disk system).

From my experience - sometimes it's better to use lzma(-2 for example) to compress/decompress files - it have higher compression than gzip so much less data needs to be read from disk and speed is comparable.

|improve this answer|||||
  • 1
    Indeed, it's my case. Very CPU hungry Java process is used instead of grep. I've simplify question a little bit. And still, parallel eating a lot of CPU doesn't provide a lot of work to Java processes. – Denis Bazhenov Feb 3 '15 at 10:44
1

The decompression is the bottleneck here. If decompression is not parallelized internally, you won't achieve it by yourself. If you have more than one job like that, then of course launch them in parallel, but your pipeline by itself is hard to parallelize. Splitting one stream into parallel streams is almost never worth it, and can be very painful with synchronization and merging. Sometimes you just have to accept that multiple cores won't help with every single task you are running.

In general, parallelization in shell should mostly be on the level of independent processes.

|improve this answer|||||
  • 1
    It doesn't seems like decompression is bottleneck in case of using parallel. I agree that it certainly is in the first case (w/o parallel), but in the second one (with parallel) bottleneck is on the parallel side. This follows from observation that throughput is drops down significantly as measured by pv. If bottleneck is in decompression, throughput will not change whatever you add to the pipeline. It's very intuitive definition of throughput, I guess – the thing that limits the throughput the most. – Denis Bazhenov Feb 3 '15 at 11:15
  • 1
    It is possible that grep is so fast, that it finishes faster than parallel can write to its pipe. In this case, most grep processes simply wait to get more, while parallel is working around the clock to multiplex the blocks into several pipes (which are extra IO operations and may even block decompression if the buffer is full). Did you also try to play with the --block parameter? It defaults to 1M so until one grep gets 1M data, the rest are almost certainly finished already. Therefore we are back to the fact that it makes no sense to parallelize this. – orion Feb 3 '15 at 11:25
  • 1
    Yep, I have tried this options with large and small block size. As well as different values for -N/-L options. It seems like default options are very close to local optimum I've experiencing :) – Denis Bazhenov Feb 3 '15 at 11:29
  • 1
    Try timing it with and without pv (with time). This way you can see if pv itself is slowing it down. If it is, then parallel copying data into pipes is definitely additional overhead. And in any case, I'm quite sure that, grep is almost real-time in this case, especially if the pattern is a simple string without much backtracking. Additionally, parallel will interleave and mess up the grep outputs. – orion Feb 3 '15 at 11:39
  • 1
    I will crosscheck that pv itself doesn't cause the problem, thank you for the advice. – Denis Bazhenov Feb 3 '15 at 12:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.