Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. It only takes a minute to sign up.
Sign up to join this community
I have a tar.gz file which contains a few million files in it. I want to divide the contents without decompressing it due to a cap on the number of files. Is there a way for me to divide a tar/tar.gz file in multiple tar/tar.gz.
if it is not possible, can it be done in python instead?
1st post let me know if I did something wrong.
Edit: I don't want to put it back together. I want to be able to work on each 'splitted' files.
You can split it with split, but the individual pieces won't make any sense until put back together. If you want to split it into self-contained archives, you will need to decompress it.
tar.gz are compressed as streams of bytes and the compression smushes everything together. There's no way to get the file boundaries back without decompression.
split is the right tool for it.
To split a file in 10MB chunks
split -b10m /path/to/file parts
To put it together
cat parts* > file
tar -c -M -L {NUMBER x 1024 bytes} -f part1.tar large.tar
Jul 3, 2016 at 21:04
.tar.gz file is not a .zip file. You probably want python-libarchive-c instead of zipfile
It is not possible to split the compressed tar archive into pieces without decompression: the compression is applied on top of the 'tar stream' and this stream is treated by a compressor as an opaque binary stream.
Thus, any tool that could produce a set of tar files from your original compressed tar archive will actually decompress it.
The tar itself has no mode that takes a tar archive as an input and produces another tar as an output file apart from append and delete modes.
As your tar archive is pretty big, you probably don't want to decompress it several times to extract a files portion by portion while creating separate archive for each set of extracted files.
There is a python tar library that probably allows 'in flight' tar file processing (I am not as strong in python to verify this myself). So you could develop a python script that would read items from the source archive and then append them to the current output archive until its size reaches a limit of your choice. Then, the script would create the next output archive and proceed further.
tarcombines a bunch of files into one big file.gzipcompresses the big file into a little file. Decompressing the one little file gives you one big file; extracting the one big file gives you a bunch of files.arjarchiver, viaapt install arj.