Can tarring files improve compression?

Question

Can tarring a bunch of files together improve compression with the standard tools, e.g. gzip, bzip2, xz?

I've long thought this to be the case but never tested it out. If we have 2 copies of the same 20Mb file of random bytes tarred together, a clever compression program that realizes this could compress the entire tarball down to almost 20Mb.

I just tried this experiment using gzip, bzip2, and xz to compress 1) a file of random bytes, 2) a tarball of two copies of that file, and 3) a cat of two copies of that file. In all cases the compression did not reduce the file size. This is expected for case 1 but for cases 2 and 3 the optimal result is that a 40Mb file can be shrunk to nearly 20Mb. That's a difficult insight for a compression program to see, especially because the redundancy is distant, so I wouldn't expect a perfect result but I still had figured there would be some compression.

Test:

dd if=/dev/urandom of=random1.txt bs=1M count=20
cp random1.txt random2.txt
cat random1.txt random2.txt > random_cat.txt
tar -cf randoms.tar random1.txt random2.txt
gzip -k random* &
bzip2 -k random* &
xz -k random* &
wait
du -sh random*

Result:

20+0 records in
20+0 records out
20971520 bytes (21 MB) copied, 1.40937 s, 14.9 MB/s
[1]   Done                    gzip -k random*
[2]-  Done                    bzip2 -k random*
[3]+  Done                    xz -k random*
20M random1.txt
21M random1.txt.bz2
21M random1.txt.gz
21M random1.txt.xz
20M random2.txt
21M random2.txt.bz2
21M random2.txt.gz
21M random2.txt.xz
40M random_cat.txt
41M random_cat.txt.bz2
41M random_cat.txt.gz
41M random_cat.txt.xz
41M randoms.tar
41M randoms.tar.bz2
41M randoms.tar.gz
41M randoms.tar.xz

Is this generally what I should expect?

Is there a way to improve compression here?

Answer 1

You're up against the "block size" of the compressor. Most compression programs break the input into blocks and compress each block. It appears the bzip block size only goes up to 900K, so it won't see any pattern that takes longer than 900K bytes to repeat.

http://www.bzip.org/1.0.3/html/memory-management.html

gzip appears to use 32K blocks.

With xz you're in luck though! From the man page:

   Preset   DictSize   CompCPU   CompMem   DecMem
     -0     256 KiB       0        3 MiB    1 MiB
     -1       1 MiB       1        9 MiB    2 MiB
     -2       2 MiB       2       17 MiB    3 MiB
     -3       4 MiB       3       32 MiB    5 MiB
     -4       4 MiB       4       48 MiB    5 MiB
     -5       8 MiB       5       94 MiB    9 MiB
     -6       8 MiB       6       94 MiB    9 MiB
     -7      16 MiB       6      186 MiB   17 MiB
     -8      32 MiB       6      370 MiB   33 MiB
     -9      64 MiB       6      674 MiB   65 MiB

so "xz -8" will find up to 32MB patterns, and "xz -9" up to 64MB patterns. But beware how much ram it is requiring to perform the compression (and to decompress)...

Answer 2

As already indicated:

1. Using random files is not good since they already contain maximum "information entropy", therefore won't compress;
2. You need to pack a lot of files for a fair comparison.

A better test case might be this:

cd /var/tmp
tar -zcf test1.tar /usr
tar -cf test2.tar /usr
gzip test2.tar
ls -h

(Note: Hoping there are no mounts under /usr!)

You can use tar -jcf for xz compression instead.

Now if test2.tar.gz is smaller than test1.tar.gz, then the test is successful (i.e. tarring files then compressing is better than compressing then tarring). My guess is it will be, for a lot (i.e. thousands) of files. The downside is it will potentially take longer to execute, as well as requiring a lot more disk space, since it has to build the entire tar file first and then compress it. That's why the 1st method is often used instead, as it compresses each file on the fly, even though it may not give as small a tarball.

For example, in our offsite backup we're typically backing up 4,000,000 files totaling about 2TB. So the first method is far quicker and doesn't require an additional 2TB of disk.

Answer 3

The random file content you chose is not a good example - the compressed tarfiles will be bigger than the originals. You'll see the same with files in already compressing formats (many image/audio/video formats, for example).

But tar-ing together multiple files with compressible content would typically produce smaller total tarfile size than when tar-ing them separately, especially when the contents are similar (for example logfiles from the same program). The reason is that some of the per-file compression offset data (like pattern arrays for some compression algorithms) could be shared by all files in the same tarfile.