Compress a large folder tree while keeping an easy way to navigate and extract particular files

Question

I have a large and heavy folder tree with many files and folders. I want to compress it to reduce the storage size. This tree will not be frequently used any more, but quite probably I will need to navigate it in the future and extract one or two specific files from it.

Thus, I do not want to have a tremendous large compressed tar file with the full tree because this will make very difficult for navigate it later and I will have to decompress everything to extract any file... Right?. It also takes twice the space until I get the tar file ready and then I can remove the original tree.

So I was thinking to use something like bz2 or xz and compress each file separately. However, if I have many many small and similar files in a sub-folder, I guess that compress each file will be less optimal than aggregate all files in a tar file and then compress it... Right? (I am assuming that the compressor can find more redundancy if there is more information to compress).

Thus, what should I do?

The idea to aggregate the files to get the best compression is opposed to the idea to keep them separately in order to later navigate and extract the information easily. Is there a tool, or a recommended way, to choose a middle point? For instance, should I search all the end-folders (the ones at the end of each branch of the tree) and first make a tar of their contents and then compress them, and then move on to the above depth and iterate?

EDIT: test case

I made a comparison between the compression of individual files with xz and the compression of everything with 7z. The test case consist of a folder tree of 58000.66 M with 1250397 files sorted into 4290 directories. After compressing everything with:

find . -type f -print0 | xargs -0 xz

I get 14576.68 M. However, if I compress the entire tree with 7z I get 9622.74 M, almost 5 Gb less.

Answer 1

...have to decompress everything to extract any file... Right?.

not if you avoid tar; searching file names, and extracting a single file from an archive is easy (and fast with .zip, .7z). Example;

> du -h a.*                     
223M    a                                        
115M    a.tar.gz                
75M     a.7z            

> 7z l a.7z | wc -l                     
2253  

> ( time 7z l a.7z >/dev/null ) 2>&1 | grep user                                            
user    0m0.014s

> ( time tar -ztf a.tar.gz >/dev/null ) 2>&1 | grep user
user    0m2.055s

...compress each file will be less optimal than aggregate all files in a tar file and then compress it... Right?

Correct.

My first comment is that 7z is multi-threaded and indexed, where as tar.xz is not so there is a huge performance difference.

But really I would just use a file system or device mapper for compression;

vdo create --name=vdo_volume --device=/dev/vda

File systems that compress; https://en.m.wikipedia.org/wiki/Category:Compression_file_systems eg

mount -t btrfs  -o compress=lzo /dev/sdb /media/my_compressed_files

Note that both of these can be done on a loop-back device so they can act like an entire file system in a file.

There are also archive fuse drivers like fuse-7z-ng which would be fast for data retrieval but the write performance of those would be horrendous.

fuse-7z-ng files.7z /media/my_compressed_files

Answer 2

With tar's -tvf options, you can list the contents of any archive created with tar, even in combinations with other libraries like gzip, bzip2, or xz.

tar -tf file.tar
tar -ztf file.tar.gz
tar -jtf file.tar.bz2
tar -Jtf file.tar.xz

Listing the contents allows you to specifically identify the file you want to extract from the achive, and then you can use the specific path to extract the file.

Let's say your path is home/user/old/photos/beach2012/bigbeachball.jpg.

tar -xf file.tar home/user/old/photos/beach2012/bigbeachball.jpg
tar -zxf file.tar.gz home/user/old/photos/beach2012/bigbeachball.jpg
tar -jxf file.tar.bz2 home/user/old/photos/beach2012/bigbeachball.jpg
tar -Jxf file.tar.xz home/user/old/photos/beach2012/bigbeachball.jpg

If your archive file isn't going to change much (ie; you're not going be be adding or deleting files to/from it very often), you could always extract the contents into a text file. Having the contents in a text file you can search with grep could make finding files easier than having to process the archive each time you want to search.

Answer 3

I've always been really interested in this, here are a couple of options I've worked on:

squashfs

Squashfs lets you archive and compress data, but then later mount it as a filesystem so you can browse it as if it was uncompressed:

mksquashfs some/directory dir.squashfs
mkdir mnt
sudo mount -t squashfs dir.squashfs mnt 

This only works as root, and only on Linux. But squashfuse allows you to do this as non-root on any FUSE-supporting system.

pixz

Someone already mentioned pixz--but it's not just a parallelized xz. It also adds a file index to the compressed archive, that makes small operations much more efficient. For example:

# Listing files, and extracting a single file, using normal tar + xz
# So slow, not practical for interactive use.

$ time tar -tf 8gigs.tpxz > /dev/null
371.99s user 5.45s system 99% cpu 6:21.00 total
$ time tar -xf 8gigs.tpxz dir/somefile.txt
375.04s user 5.45s system 99% cpu 6:21.00 total

# Using pixz instead it's much faster!

$ time pixz -l < 8gigs.tpxz  > /dev/null
0.01s user 0.01s system 38% cpu 0.035 total
$ time pixz -x dir/somefile.txt < 8gigs.tpxz | tar x
0.33s user 0.02s system 97% cpu 0.359 total

Disclaimers

I wrote pixz and squashfuse--because I had problems like yours.