Can tarring a bunch of files together improve compression with the standard tools, e.g. gzip, bzip2, xz?
I've long thought this to be the case but never tested it out. If we have 2 copies of the same 20Mb file of random bytes tarred together, a clever compression program that realizes this could compress the entire tarball down to almost 20Mb.
I just tried this experiment using gzip, bzip2, and xz to compress 1) a file of random bytes, 2) a tarball of two copies of that file, and 3) a cat of two copies of that file. In all cases the compression did not reduce the file size. This is expected for case 1 but for cases 2 and 3 the optimal result is that a 40Mb file can be shrunk to nearly 20Mb. That's a difficult insight for a compression program to see, especially because the redundancy is distant, so I wouldn't expect a perfect result but I still had figured there would be some compression.
dd if=/dev/urandom of=random1.txt bs=1M count=20 cp random1.txt random2.txt cat random1.txt random2.txt > random_cat.txt tar -cf randoms.tar random1.txt random2.txt gzip -k random* & bzip2 -k random* & xz -k random* & wait du -sh random*
20+0 records in 20+0 records out 20971520 bytes (21 MB) copied, 1.40937 s, 14.9 MB/s  Done gzip -k random* - Done bzip2 -k random* + Done xz -k random* 20M random1.txt 21M random1.txt.bz2 21M random1.txt.gz 21M random1.txt.xz 20M random2.txt 21M random2.txt.bz2 21M random2.txt.gz 21M random2.txt.xz 40M random_cat.txt 41M random_cat.txt.bz2 41M random_cat.txt.gz 41M random_cat.txt.xz 41M randoms.tar 41M randoms.tar.bz2 41M randoms.tar.gz 41M randoms.tar.xz
Is this generally what I should expect?
Is there a way to improve compression here?