3

I have a tar.gz file which contains a few million files in it. I want to divide the contents without decompressing it due to a cap on the number of files. Is there a way for me to divide a tar/tar.gz file in multiple tar/tar.gz.

if it is not possible, can it be done in python instead?

1st post let me know if I did something wrong.

Edit: I don't want to put it back together. I want to be able to work on each 'splitted' files.

|improve this question
  • "without decompressing it due to a cap on the number of files." - you don't want to decompress it, or you don't want to extract it? – immibis Jul 3 '16 at 23:17
  • I don't want to decompress it. as the cap(number of files I am allowed) << files compressed. – Joshua Cabanas Jul 4 '16 at 0:28
  • tar combines a bunch of files into one big file. gzip compresses the big file into a little file. Decompressing the one little file gives you one big file; extracting the one big file gives you a bunch of files. – immibis Jul 4 '16 at 0:33
  • thanks for clarifying, not really well versed with the difference of decompress/extract in this context. I don't want to extract the contents of the tar file. – Joshua Cabanas Jul 4 '16 at 0:37
2

You can split it with split, but the individual pieces won't make any sense until put back together. If you want to split it into self-contained archives, you will need to decompress it.

|improve this answer
  • I want it into a self-contained archives. – Joshua Cabanas Jul 3 '16 at 20:46
  • @JoshuaCabanas You're out of luck. You'll need to decompress first. tar.gz are compressed as streams of bytes and the compression smushes everything together. There's no way to get the file boundaries back without decompression. – PSkocik Jul 3 '16 at 20:49
1

split is the right tool for it.

To split a file in 10MB chunks

split -b10m /path/to/file parts

To put it together

cat parts* > file

|improve this answer
1

It is not possible to split the compressed tar archive into pieces without decompression: the compression is applied on top of the 'tar stream' and this stream is treated by a compressor as an opaque binary stream.

Thus, any tool that could produce a set of tar files from your original compressed tar archive will actually decompress it.

The tar itself has no mode that takes a tar archive as an input and produces another tar as an output file apart from append and delete modes.

As your tar archive is pretty big, you probably don't want to decompress it several times to extract a files portion by portion while creating separate archive for each set of extracted files.

There is a python tar library that probably allows 'in flight' tar file processing (I am not as strong in python to verify this myself). So you could develop a python script that would read items from the source archive and then append them to the current output archive until its size reaches a limit of your choice. Then, the script would create the next output archive and proceed further.

|improve this answer
  • I am actually working with the python library 'tarfile' which does exactly as you said. However, I am trying to process each file in the tar archive and trying splitting it, so that I can run multiple instance of my program. – Joshua Cabanas Jul 3 '16 at 21:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.