I have a large and heavy folder tree with many files and folders. I want to compress it to reduce the storage size. This tree will not be frequently used any more, but quite probably I will need to navigate it in the future and extract one or two specific files from it.
Thus, I do not want to have a tremendous large compressed tar file with the full tree because this will make very difficult for navigate it later and I will have to decompress everything to extract any file... Right?. It also takes twice the space until I get the tar file ready and then I can remove the original tree.
So I was thinking to use something like bz2 or xz and compress each file separately. However, if I have many many small and similar files in a sub-folder, I guess that compress each file will be less optimal than aggregate all files in a tar file and then compress it... Right? (I am assuming that the compressor can find more redundancy if there is more information to compress).
Thus, what should I do?
The idea to aggregate the files to get the best compression is opposed to the idea to keep them separately in order to later navigate and extract the information easily. Is there a tool, or a recommended way, to choose a middle point? For instance, should I search all the end-folders (the ones at the end of each branch of the tree) and first make a tar of their contents and then compress them, and then move on to the above depth and iterate?
EDIT: test case
I made a comparison between the compression of individual files with
xz and the compression of everything with
7z. The test case consist of a folder tree of 58000.66 M with 1250397 files sorted into 4290 directories. After compressing everything with:
find . -type f -print0 | xargs -0 xz
I get 14576.68 M. However, if I compress the entire tree with
7z I get 9622.74 M, almost 5 Gb less.