I have a file I need to split into smaller sizes (<24M when zipped)

Heres the file:

498775505 Mar  8 00:08 test.file

I split it:

split -b 125000k test.file test.file.

Now I have even sized files (apart from the last file which is fine)

476M Mar  8 00:08 test.file
123M Mar  8 00:09 test.file.aa
123M Mar  8 00:09 test.file.ab
123M Mar  8 00:09 test.file.ac
110M Mar  8 00:09 test.file.ad

But when I gzip these files, they do not zip down evenly

gzip test.file.a*

476M Mar  8 00:08 test.file
27M Mar  8 00:09 test.file.aa.gz
23M Mar  8 00:09 test.file.ab.gz
22M Mar  8 00:09 test.file.ac.gz
20M Mar  8 00:09 test.file.ad.gz

Can somebody explain what is happening here with gzip?

(This is more out of curiosity as I can just split them into smaller amounts to get them under 24M, just wondering how gzip works here)

  • Do all parts have the same contents? If not, why should the compress to the same size?
    – muru
    Mar 8 '18 at 0:24
  • Your requirement is not very clear but you should instead compress the whole file, and split the results in smaller pieces. Each piece will be useless just by itself, you will need all of them back again together, but your current solution is in same case and will produce worse results compression wise. Mar 8 '18 at 0:38

The split files contain different parts of the original (full) file, they probably have different contents. (The only way they would be identical would be for the original to be highly repetitive.)

Different contents result in different compression results. Stuff like aaaaaaaaaa is easier to compress than wekfsiorlm. In files of 123 MB, there's quite a lot of space for one file to be more "random"-looking (more difficult to compress) than another, even if it's not as extreme as my example here.

If you want to control the sizes of the compressed result files, you could split the original into smaller pieces, compress them individually, and then concatenate the compressed parts together, up to the desired size limit. (I can't think of a trivial way to do that, though.)

If the input to gzip -d contains multiple compressed gzip "files", it decompresses them all. Though this would lose some compression performance, since the splitting causes artificial breaks to the data.

  • 1
    Or compress the original file and split that, which would require concatenating the bits together in the correct order before uncompressing.
    – Kusalananda
    Mar 8 '18 at 6:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.