1

A long time ago, before I tried incremental and differential backups, I tried to tar/gzip several similar large (1 GB) directories, but they did not compress any better than tarring and gzipping each directory individually. My guess why it didn't work is this:

  1. tar was likely not going to put duplicate files next to each other

  2. because files were far away, they would be in separate gzip DEFLATE blocks and so not be compressed together (I've also asked how far)

Is this reasoning correct?

Improve this question

1 Answer 1

Reset to default
2

Yes, your reasoning is correct, as tar doesn't sort files by extensions (which could have helped a lot to achieve higher compression ratios) and gzip is a very old compression algorithm with a relatively modest dictionary, just 32KB.

Please try using xz or p7zip instead.

Here's a compression string which allows me to achieve the highest compression ratio under Linux:

7za a -mx=9 -myx=9 -mfb=273 -bt -slp -mmt4 -md=1536m -mqs archive.7z [list of files]

This requires a ton of memory (at the very least 32GB of RAM). If you remove -mmmt4 and reduce the dictionary size to say 1024m, 16GB would be enough.

Speaking of sorting files for tar. I wrote a script which does just that a few years ago: https://github.com/birdie-github/useful-scripts/blob/master/tar_sorted

Improve this answer
2
  • Would the hypothetical best tar solution be some way of getting tar to put identical files (in different locations) next to each other? Maybe by timestamp or hash?
    – qwr
    Jul 10, 2020 at 6:18
  • @qwr I've edited the answer and added a link to the script which sorts files by extension which should help improve the compression ratio quite a lot. As for timestamps or hashes - that solely depends on how you've added files in the first place. Putting files with the same hash next to each other will only help if they are relatively small, as again, gzip has a very small dictionary.
    – Artem S. Tashkinov
    Jul 10, 2020 at 6:25

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .