Tell me more ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.
up vote 3 down vote favorite

I've got a few full old backups of things like binary database dumps. Obviously, they don't differ much so doing full backups is not the smartest idea here. For now, I'm looking for a compression program capable of taking an advantage from the fact, that most of the files have a similar content.

share|improve this question
Tried lrzip? ck.kolivas.org/apps/lrzip/README – Mat Jul 15 '12 at 11:06
2  
Obnam is a backup tool featuring de-duplication (see this LWN article). – sr_ Jul 15 '12 at 12:16
Not sure I entirely understood the question but does rdiff-backup sound useful? – jw013 Aug 16 '12 at 21:23
@Mat: In the meantime I've tried lrzip, but it was extremely slow (many hours for some 100 GB). However, there might have been a problem with the NTFS partition the data were on. – maaartinus Aug 16 '12 at 21:38
@jw013: I don't think rdiff-backup would do, as it's efficient with slowly evolving files, while all I have now is a set of old snapshots. Things like obnam or bup would help (but the former requires a newer system while the latter has still problems with removal from the backup). – maaartinus Aug 16 '12 at 21:43

1 Answer

up vote 2 down vote

If you first tar the files (using tar cvf my_backup.tar <file list...>) then any compression tool will do a good job as they will see the data as one big file.

So just tar the files, and then put them in a zip, 7-zip, bzip2, etc. From the tar file, you can try the different compression algorithm and see which one performs best.

share|improve this answer
Right, you be sure to use a format that archives and then compresses, instead of one (like .zip) which compresses each file individually and then archives them – lk- Aug 16 '12 at 18:00
tar + lzma seems to work really well for binary blobs. LZMA is built into newer versions of tar too. – Chris S Aug 16 '12 at 18:17
@Huygens: AFAIK you're wrong. All compression programs I know work with some window (typically 64 kB) and see nothing beyond it as stated e.g. here on page 7. – maaartinus Aug 16 '12 at 21:46
@maaartinus this is a shortcoming of gzip then, not tar. Try another compression algorithm, and try to use the options. To experiment you can tar twice the same file and compress the resulting output and see the size as compare to compressing only one file. – Huygens Aug 20 '12 at 15:06
@Huygens You can't expect a compression algorithm go checking if (some prefix of) a 100 MiB chunk appears again in the input. I'd just suggest burning the whole lot to DVDs or such, and hope they won't be needed... – vonbrand Jan 16 at 10:52

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.