I have a big .gz file. I would like to split it into 100 smaller gzip files, that can each be decompressed by itself. In other words: I am not looking for a way of chopping up the .gz file into chunks that would have to be put back together to be able to decompress it. I want to be able to decompress each of the smaller files independently.

Can it be done without recompressing the whole file?

Can it be done if the original file is compressed with --rsyncable? ("Cater better to the rsync program by periodically resetting the internal structure of the compressed data stream." sounds like these reset points might be good places to split at and probably prepend a header.)

Can it be done for any of the other compressed formats? I would imagine bzip2 would be doable - as it is compressed in blocks.

  • 1
    Have you try split -b ? – George Vasiliou Jan 17 '17 at 23:24
  • 4
    @GeorgeVasiliou It will not result in smaller gzip files that can be decompressed. – Ole Tange Jan 17 '17 at 23:39
  • The answer to your first question is no, this has been covered in Delete last line of gz file. The answer is probably no with most compressed formats, since what you're asking for goes against compression. I think the answer is also no with gzip --rsyncable given that “gunzip cannot tell the difference” (if you could find a place to split, you could tell that there is a place to split). It might be doable with bzip2 because of its peculiar block feature. – Gilles 'SO- stop being evil' Jan 17 '17 at 23:58
  • This may help: stackoverflow.com/a/22628945/4941495 Just let the standard input stream be the output of gzip -d -c bigfile.gz. – Kusalananda Jan 18 '17 at 0:18
  • Without recompressing, it would be doable with a bzip2 file indeed. It would be doable with gz or xz only by compressing each chunk independently, so this would require a recompression. – xhienne Jan 18 '17 at 0:59

Split and join of the big file works, but it is impossible to decompress pieces of the compressed file, because essential informations are distributed through the whole dataset. Another way; split the uncompressed file and compress the single parts. Now you can decompress each pieces. But why? You have to merge all decompressed parts before further processing.

| improve this answer | |
  • 1
    Fun fact: When you have the individually compressed parts (using gzip or xz), you may do concatenation and decompression, or decompression and concatenation. The order doesn't matter. – Kusalananda Feb 24 '17 at 9:13
  • Maybe, it depends on the data. If you split and compress disk images, you have a chance to recover parts of the filesystem. If you first compress and then split, you have definitively no chance. – ingopingo Feb 24 '17 at 9:24
  • No, and that was not my premise either. I just said that the order in which you do concatenation and decompression when you have individually compressed parts does not matter (this is due to the compressed file formats). If compressing first, then splitting, then one obviously need to recombine first. – Kusalananda Feb 24 '17 at 9:26
  • Oh that's cool. It work's, even though every part contains a individual file header! – ingopingo Feb 24 '17 at 9:36

Except my error, i think this is not possible, without alterate your file in losing the ability to rebuild and decompress the big file because you will lose the metadata (header and tail) from the first big file compress and those metadata don't exist for each of your small file.

But you could create a wrapper that could do...

  1. (optional) compress the big file
  2. split your big file into 100 small chunk
  3. compress each of your small chunck in gzip
  4. decompress each chunck in gzip.
  5. concat chunck into the big file.
  6. (optional) decompress the big file

Notice : I am not sure in terme of your purpose... save storage ? save time network transmission ? limite space system ? what is your root need ?

Best Regards

| improve this answer | |
  • I want to take a gzip file and decompress it in parallel. And before you say pigz please test how well DEcompressing works on a 64-core machine. – Ole Tange Oct 27 at 22:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.