I have a tar.gz file which contains a few million files in it. I want to divide the contents without decompressing it due to a cap on the number of files. Is there a way for me to divide a tar/tar.gz file in multiple tar/tar.gz.

if it is not possible, can it be done in python instead?

1st post let me know if I did something wrong.

Edit: I don't want to put it back together. I want to be able to work on each 'splitted' files.

  • "without decompressing it due to a cap on the number of files." - you don't want to decompress it, or you don't want to extract it?
    – user253751
    Jul 3, 2016 at 23:17
  • I don't want to decompress it. as the cap(number of files I am allowed) << files compressed. Jul 4, 2016 at 0:28
  • tar combines a bunch of files into one big file. gzip compresses the big file into a little file. Decompressing the one little file gives you one big file; extracting the one big file gives you a bunch of files.
    – user253751
    Jul 4, 2016 at 0:33
  • thanks for clarifying, not really well versed with the difference of decompress/extract in this context. I don't want to extract the contents of the tar file. Jul 4, 2016 at 0:37
  • Try using arj archiver, via apt install arj.
    – Vascon63
    Feb 20, 2020 at 18:33

3 Answers 3


split is the right tool for it.

To split a file in 10MB chunks

split -b10m /path/to/file parts

To put it together

cat parts* > file

  • I don't want to put it back together I want to be able to work on each 'splitted' files. Jul 3, 2016 at 20:39
  • with gnu-tar you can repackage it with tar -c -M -L {NUMBER x 1024 bytes} -f part1.tar large.tar Jul 3, 2016 at 21:04
  • to work with python you can use the included zip lib . have a look here : [link][stackoverflow.com/questions/19371860/… Jul 3, 2016 at 21:10
  • A .tar.gz file is not a .zip file. You probably want python-libarchive-c instead of zipfile
    – cas
    Jul 4, 2016 at 1:33

You can split it with split, but the individual pieces won't make any sense until put back together. If you want to split it into self-contained archives, you will need to decompress it.

  • I want it into a self-contained archives. Jul 3, 2016 at 20:46
  • @JoshuaCabanas You're out of luck. You'll need to decompress first. tar.gz are compressed as streams of bytes and the compression smushes everything together. There's no way to get the file boundaries back without decompression. Jul 3, 2016 at 20:49

It is not possible to split the compressed tar archive into pieces without decompression: the compression is applied on top of the 'tar stream' and this stream is treated by a compressor as an opaque binary stream.

Thus, any tool that could produce a set of tar files from your original compressed tar archive will actually decompress it.

The tar itself has no mode that takes a tar archive as an input and produces another tar as an output file apart from append and delete modes.

As your tar archive is pretty big, you probably don't want to decompress it several times to extract a files portion by portion while creating separate archive for each set of extracted files.

There is a python tar library that probably allows 'in flight' tar file processing (I am not as strong in python to verify this myself). So you could develop a python script that would read items from the source archive and then append them to the current output archive until its size reaches a limit of your choice. Then, the script would create the next output archive and proceed further.

  • I am actually working with the python library 'tarfile' which does exactly as you said. However, I am trying to process each file in the tar archive and trying splitting it, so that I can run multiple instance of my program. Jul 3, 2016 at 21:47

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .