0

I have a large and heavy folder tree with many files and folders. I want to compress it to reduce the storage size. This tree will not be frequently used any more, but quite probably I will need to navigate it in the future and extract one or two specific files from it.

Thus, I do not want to have a tremendous large compressed tar file with the full tree because this will make very difficult for navigate it later and I will have to decompress everything to extract any file... Right?. It also takes twice the space until I get the tar file ready and then I can remove the original tree.

So I was thinking to use something like bz2 or xz and compress each file separately. However, if I have many many small and similar files in a sub-folder, I guess that compress each file will be less optimal than aggregate all files in a tar file and then compress it... Right? (I am assuming that the compressor can find more redundancy if there is more information to compress).

Thus, what should I do?

The idea to aggregate the files to get the best compression is opposed to the idea to keep them separately in order to later navigate and extract the information easily. Is there a tool, or a recommended way, to choose a middle point? For instance, should I search all the end-folders (the ones at the end of each branch of the tree) and first make a tar of their contents and then compress them, and then move on to the above depth and iterate?

EDIT: test case

I made a comparison between the compression of individual files with xz and the compression of everything with 7z. The test case consist of a folder tree of 58000.66 M with 1250397 files sorted into 4290 directories. After compressing everything with:

find . -type f -print0 | xargs -0 xz

I get 14576.68 M. However, if I compress the entire tree with 7z I get 9622.74 M, almost 5 Gb less.

  • 1
    have you considered using a filesystem that supports transparent "on-the-fly" compression? e.g. btrfs or zfs? these both support many other features (including snapshots, error-checking-and-correction, and much more) but the compression is very useful even if you don't make much use of the other features. – cas Aug 1 at 0:19
  • 2
    BTW, these won't give as good a compression ratio as bz2 or xz - they use faster compression methods like lz4 - but for highly-compressible data like plain text, it's good enough. – cas Aug 1 at 0:20
3

...have to decompress everything to extract any file... Right?.

not if you avoid tar; searching file names, and extracting a single file from an archive is easy (and fast with .zip, .7z). Example;

> du -h a.*                     
223M    a                                        
115M    a.tar.gz                
75M     a.7z            

> 7z l a.7z | wc -l                     
2253  

> ( time 7z l a.7z >/dev/null ) 2>&1 | grep user                                            
user    0m0.014s

> ( time tar -ztf a.tar.gz >/dev/null ) 2>&1 | grep user
user    0m2.055s

...compress each file will be less optimal than aggregate all files in a tar file and then compress it... Right?

Correct.

My first comment is that 7z is multi-threaded and indexed, where as tar.xz is not so there is a huge performance difference.

But really I would just use a file system or device mapper for compression;

vdo create --name=vdo_volume --device=/dev/vda

File systems that compress; https://en.m.wikipedia.org/wiki/Category:Compression_file_systems eg

mount -t btrfs  -o compress=lzo /dev/sdb /media/my_compressed_files

Note that both of these can be done on a loop-back device so they can act like an entire file system in a file.

There are also archive fuse drivers like fuse-7z-ng which would be fast for data retrieval but the write performance of those would be horrendous.

fuse-7z-ng files.7z /media/my_compressed_files
  • 1
    Once you created the archive, a fuse mount would be an interesting way to search it and copy out files. – Tim Kennedy Jul 31 at 20:07
  • There are multi-core/thread versions of xz available. e.g. pixz - pixz compresses and decompresses files using multiple processors. If the input looks like a tar(1) archive, it also creates an index of all the files in the archive – cas Aug 1 at 0:23
  • @cas but pixz would still be slow because of the tar limitations (see example in answer) – user1133275 Aug 1 at 1:29
  • 1
    7z is not a good tool for backing up unix files, because it does not store owner, group (and iirc permissions) of the file. The 7z man page points this out, under "Backup and limitations". But my only point was that there is a parallel version of xz. In any case, whether it's tar or 7z or cpio or zip or whatever, it's the compression that really benefits from many cores/threads, not reading the files - that's always going to be IO limited. – cas Aug 1 at 1:39
  • 1
    @cas yes I assume no users care about the 7z limitations but would care a lot for an index listing to be instant instead of bound to the archive size. (zip and 7z onle read the tiny index, where as tar reads the whole archive to list contents) – user1133275 Aug 1 at 1:49
1

With tar's -tvf options, you can list the contents of any archive created with tar, even in combinations with other libraries like gzip, bzip2, or xz.

tar -tf file.tar
tar -ztf file.tar.gz
tar -jtf file.tar.bz2
tar -Jtf file.tar.xz

Listing the contents allows you to specifically identify the file you want to extract from the achive, and then you can use the specific path to extract the file.

Let's say your path is home/user/old/photos/beach2012/bigbeachball.jpg.

tar -xf file.tar home/user/old/photos/beach2012/bigbeachball.jpg
tar -zxf file.tar.gz home/user/old/photos/beach2012/bigbeachball.jpg
tar -jxf file.tar.bz2 home/user/old/photos/beach2012/bigbeachball.jpg
tar -Jxf file.tar.xz home/user/old/photos/beach2012/bigbeachball.jpg

If your archive file isn't going to change much (ie; you're not going be be adding or deleting files to/from it very often), you could always extract the contents into a text file. Having the contents in a text file you can search with grep could make finding files easier than having to process the archive each time you want to search.

  • 1
    But tar is slow O(n) and 7z is fast O(1) for content listing. (see example in my answer) – user1133275 Aug 1 at 1:39
  • that is accurate, and that's why i mentioned listing the contents once and saving it to a text file. OP mentioned that it would likely be accessed very infrequently, though, and he didn't mention how big it was, and tar is a very tried and true for long term storage of archives. I don't have the same confidence in 7z for long term archival (just from personal experience). – Tim Kennedy Aug 1 at 11:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.