I've got a few full old backups of things like binary database dumps. Obviously, they don't differ much so doing full backups is not the smartest idea here. For now, I'm looking for a compression program capable of taking an advantage from the fact, that most of the files have a similar content.

  • Tried lrzip? ck.kolivas.org/apps/lrzip/README
    – Mat
    Jul 15, 2012 at 11:06
  • 2
    Obnam is a backup tool featuring de-duplication (see this LWN article).
    – sr_
    Jul 15, 2012 at 12:16
  • Not sure I entirely understood the question but does rdiff-backup sound useful?
    – jw013
    Aug 16, 2012 at 21:23
  • @Mat: In the meantime I've tried lrzip, but it was extremely slow (many hours for some 100 GB). However, there might have been a problem with the NTFS partition the data were on.
    – maaartinus
    Aug 16, 2012 at 21:38
  • @jw013: I don't think rdiff-backup would do, as it's efficient with slowly evolving files, while all I have now is a set of old snapshots. Things like obnam or bup would help (but the former requires a newer system while the latter has still problems with removal from the backup).
    – maaartinus
    Aug 16, 2012 at 21:43

2 Answers 2


If you first tar the files (using tar cvf my_backup.tar <file list...>) then any compression tool will do a good job as they will see the data as one big file.

So just tar the files, and then put them in a zip, 7-zip, bzip2, etc. From the tar file, you can try the different compression algorithm and see which one performs best.

  • 1
    Right, you be sure to use a format that archives and then compresses, instead of one (like .zip) which compresses each file individually and then archives them
    – lk-
    Aug 16, 2012 at 18:00
  • tar + lzma seems to work really well for binary blobs. LZMA is built into newer versions of tar too.
    – Chris S
    Aug 16, 2012 at 18:17
  • @Huygens: AFAIK you're wrong. All compression programs I know work with some window (typically 64 kB) and see nothing beyond it as stated e.g. here on page 7.
    – maaartinus
    Aug 16, 2012 at 21:46
  • @maaartinus this is a shortcoming of gzip then, not tar. Try another compression algorithm, and try to use the options. To experiment you can tar twice the same file and compress the resulting output and see the size as compare to compressing only one file.
    – Huygens
    Aug 20, 2012 at 15:06
  • @Huygens You can't expect a compression algorithm go checking if (some prefix of) a 100 MiB chunk appears again in the input. I'd just suggest burning the whole lot to DVDs or such, and hope they won't be needed...
    – vonbrand
    Jan 16, 2013 at 10:52

I've had very good luck with 7-Zip. If you have the horsepower, it is capable of operating with a very large window. Make sure your original files are as uncompressed as possible so it can find similarities. (For Excel files in a heterogeneous environment, for example, this means unzipping their contents first, since xlsx files are lightly compressed when they're stored. I was once able to compress 600 MiB+ of almost-redundant Excel version files down to a few hundred KiB.)

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .