9

Can tarring a bunch of files together improve compression with the standard tools, e.g. gzip, bzip2, xz?

I've long thought this to be the case but never tested it out. If we have 2 copies of the same 20Mb file of random bytes tarred together, a clever compression program that realizes this could compress the entire tarball down to almost 20Mb.

I just tried this experiment using gzip, bzip2, and xz to compress 1) a file of random bytes, 2) a tarball of two copies of that file, and 3) a cat of two copies of that file. In all cases the compression did not reduce the file size. This is expected for case 1 but for cases 2 and 3 the optimal result is that a 40Mb file can be shrunk to nearly 20Mb. That's a difficult insight for a compression program to see, especially because the redundancy is distant, so I wouldn't expect a perfect result but I still had figured there would be some compression.

Test:

dd if=/dev/urandom of=random1.txt bs=1M count=20
cp random1.txt random2.txt
cat random1.txt random2.txt > random_cat.txt
tar -cf randoms.tar random1.txt random2.txt
gzip -k random* &
bzip2 -k random* &
xz -k random* &
wait
du -sh random*

Result:

20+0 records in
20+0 records out
20971520 bytes (21 MB) copied, 1.40937 s, 14.9 MB/s
[1]   Done                    gzip -k random*
[2]-  Done                    bzip2 -k random*
[3]+  Done                    xz -k random*
20M random1.txt
21M random1.txt.bz2
21M random1.txt.gz
21M random1.txt.xz
20M random2.txt
21M random2.txt.bz2
21M random2.txt.gz
21M random2.txt.xz
40M random_cat.txt
41M random_cat.txt.bz2
41M random_cat.txt.gz
41M random_cat.txt.xz
41M randoms.tar
41M randoms.tar.bz2
41M randoms.tar.gz
41M randoms.tar.xz

Is this generally what I should expect?

Is there a way to improve compression here?

  • Your test cases are bad examples. Try doing your test with, say, a directory of ~100 (real) text files. – lcd047 Jun 25 '15 at 18:08
  • Why is it a bad example? We know exactly what to expect. A random file can't be compressed and 2 of a random file can be compressed in half. – Praxeolitic Jun 25 '15 at 18:31
  • The "random" file contents are a problem. They're incompressible. Use two different large text files to get a better idea. A related idea here is "normalized compression difference". You might take a look at ims.cuhk.edu.hk/~cis/2005.4/01.pdf to see what kind of problems you can encounter doing this kind of testing. – Bruce Ediger Jun 25 '15 at 19:08
11

You're up against the "block size" of the compressor. Most compression programs break the input into blocks and compress each block. It appears the bzip block size only goes up to 900K, so it won't see any pattern that takes longer than 900K bytes to repeat.

http://www.bzip.org/1.0.3/html/memory-management.html

gzip appears to use 32K blocks.

With xz you're in luck though! From the man page:

   Preset   DictSize   CompCPU   CompMem   DecMem
     -0     256 KiB       0        3 MiB    1 MiB
     -1       1 MiB       1        9 MiB    2 MiB
     -2       2 MiB       2       17 MiB    3 MiB
     -3       4 MiB       3       32 MiB    5 MiB
     -4       4 MiB       4       48 MiB    5 MiB
     -5       8 MiB       5       94 MiB    9 MiB
     -6       8 MiB       6       94 MiB    9 MiB
     -7      16 MiB       6      186 MiB   17 MiB
     -8      32 MiB       6      370 MiB   33 MiB
     -9      64 MiB       6      674 MiB   65 MiB

so "xz -8" will find up to 32MB patterns, and "xz -9" up to 64MB patterns. But beware how much ram it is requiring to perform the compression (and to decompress)...

  • 1
    Yep, xz -8 does shrink the tarball and cat in the test to 21M. – Praxeolitic Jun 25 '15 at 17:39
  • 1
    There's more to it than just the block size. But the full story is not something that can be explained in a few paragraphs on SE. – lcd047 Jun 25 '15 at 18:10
  • 1
    @Praxeolitic A course on data compression might help. – lcd047 Jun 25 '15 at 20:03
  • 1
    @lcd047 Compression is a huge topic but the question here was simply "why didn't this compress" and the answer is because compression works on repeating patterns and the pattern he wanted it to find took longer to reoccur than any tool was looking for. – dataless Jun 28 '15 at 22:44
  • 1
    I also think its useful to know that "-9" on most command line compressors doesn't mean "try harder to find patterns", it means "consider larger pattern spaces". – dataless Jun 28 '15 at 22:51
2

The random file content you chose is not a good example - the compressed tarfiles will be bigger than the originals. You'll see the same with files in already compressing formats (many image/audio/video formats, for example).

But tar-ing together multiple files with compressible content would typically produce smaller total tarfile size than when tar-ing them separately, especially when the contents are similar (for example logfiles from the same program). The reason is that some of the per-file compression offset data (like pattern arrays for some compression algorithms) could be shared by all files in the same tarfile.

2

As already indicated:

  1. Using random files is not good since they already contain maximum "information entropy", therefore won't compress;
  2. You need to pack a lot of files for a fair comparison.

A better test case might be this:

cd /var/tmp
tar -zcf test1.tar /usr
tar -cf test2.tar /usr
gzip test2.tar
ls -h

(Note: Hoping there are no mounts under /usr!)

You can use tar -jcf for xz compression instead.

Now if test2.tar.gz is smaller than test1.tar.gz, then the test is successful (i.e. tarring files then compressing is better than compressing then tarring). My guess is it will be, for a lot (i.e. thousands) of files. The downside is it will potentially take longer to execute, as well as requiring a lot more disk space, since it has to build the entire tar file first and then compress it. That's why the 1st method is often used instead, as it compresses each file on the fly, even though it may not give as small a tarball.

For example, in our offsite backup we're typically backing up 4,000,000 files totaling about 2TB. So the first method is far quicker and doesn't require an additional 2TB of disk.

  • Doesn't -z compress the archive (ie the tar)? Usually the output filename with czf ends with .tar.gz to emphasise this. – Jari Keinänen Jun 30 '15 at 22:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.