I'm using dar (like tar, but optimized for hard disks) for backing up RAID volumes. They're modest sized for RAIDs, at perhaps 3 - 10 TB. I'm aiming to put the results on 8 TB external drives, but it would be neat to be able to fit more than one backup on a drive. This is for my personal hobby so price matters.
Anyway, I started one that's 3.4 TB with dar, using the xz:0 compression and I'm 24 hours into it and the result is only 740 GB, and I have no idea what percentage of the job has been finished. This is on a dual-xeon machine with 32 hyperthreads, but not all that fast; 4 GHz)
So I want a compressor that can handle 3 TB in a day. Better compression is nice, but right now I just want to control the amount of time it's taking. Is there a study with this kind of info? Has anybody got experience with compressing large archives?
Oh, and because of the way dar works, it has to be able to compress a stream. Dar compresses each component so it knows when to switch slices (the output may go to multiple slices on multiple disks or sticks).
