7

I have some process that creates a stream of millions of highly similar lines. I'm piping this to gz. Does the compression ratio improve over time in such a setup? I.e. is the compression ratio better for 1 million similar lines, than say 10,000?

8

It does up to a certain point and this evens out. The compression algorithms have a restriction on the size of the blocks they look at (bzip2) and/or on the tables they keep with information on previous patterns (gzip).

In the case of gzip, once a table is full old entries get pushed out, and compression no further improves. Depending on the your compression quality factor (-0 to -9) and the repetitiveness of your input this filling up can of course can take a while and you might not notice.

4

Not much. The "distance" covered by the DEFLATE algorithm which gzip uses is limited to 32 KB.

Wikipedia link -> DEFLATE

It is worth benchmarking against the various gzip compression levels and also considering bzip2 and xz.

0

Here is a overview of gzip http://www.gzip.org/algorithm.txt

Short answer is that it will not improve significantly after the initial data needed for the hashes is taken into account.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.