1

Let's say that I compress my web hosting into a tar.gz archive in order to create a backup called backup.tar.gz.

One month later I want to repeat the process. Let's assume that I'm actively developing the site so there have been a few changes since the last backup was taken.

Me (or my cron job) repeats the process a month later and I overwrite the original backup.tar.gz with the update backup archive.

If I were to use rsync to sync that backup.tar.gz to a remote backup target (say, over SFTP) would I be able to only sync the delta between the two files?

Would overwriting the file affect the behavior by creating a new timestamp?

Or would rsync be able to see into the archive, in any event, identify that most of the archive already remains on destination, and only sync over the changes?

Thank you!

2

A compressed (gzipped) file will be transformed in its entirety by the additional of just a single byte to the source. This makes it wholly unsuitable for efficient copying with rsync as even the tiniest change will require transfer of the complete file.

Fortunately, some implementations of gzip can be adjusted to compress for efficient transfer with rsync,

--rsyncable [...] With this option, rsync can transfer only the changed files as well as a small amount of metadata that is required to update the archive structure in the area that was changed.

You cannot access this flag directly from tar, so you need to use a pipeline instead of letting tar compress directly

tar cf - files and folders | gzip --rsyncable > output.tgz

(There is a GZIP environment variable that can be used to set this value for all invocations of gzip, but the documentation has it marked as obsolescent so I wouldn't advise you using it without care.)

| improve this answer | |
0

By default, Rsync only synchronises the modified blocks and bytes. Therefore, if you had synced a text file before and later added some texts to the source file when you sync, only the inserted text will be copied.

If your using a uncompressed tar file and appending file to it with

tar -rf archive.tar file3.txt

then rsync will be able to just transmit the new differences at the end of the tar file.

However, if you creating a tar file from scratch, tar in some cases can be non-deterministic in the way it add files and directories to the archive.

It's quite possible that two tar operations on very similar file systems can create vastly different tar files in terms of underlying structure.

However depending on how great this non deterministic behaviour is, the rsync delta algorithm could find some wins.

The act of compressing the output to a .gz, will only make things worse. Compressing data is act of transformation, even adding a few extra byte's to a tar file and then compressing it, can radically alter it's entire structure. Thus defeating the rolling hash algorithm that rsync implements.

in may be better not to compress the tar file it self, but allow rsync to compress the data it send with rsync, --compress or -z

Tar is not deterministic because it's add padding, it does not sort files in a particular order and it adds timestamps, etc

Some systems such as Nix/Nixos use a deterministic archive format called NAR. NAR is the Nix ARchive.

If you want to learn more about non-deterministic behaviour in archivers, and how this is overcome... you can find more in Dolstra's PhD Thesis.

https://edolstra.github.io/pubs/nixos-jfp-submitted.pdf

| improve this answer | |
  • Thank you for the detailed information. Yes, in this use case I would be generating the backup file from scratch every time with cron and not adding to a tar as you have described. This is because (in the case of web hosting) I naturally don't know and couldn't fit all the changes to the filesystem since the last build into one command. What @roaima suggested, though, (piping the output of the compression through gzip with the --rsyncable flag) seems fairly straightforward and workable. – Daniel Rosehill Jul 19 at 20:56
  • @DanielRosehill maybe you can do some benchmarking on compression rates, time to do the back and speeds. up and post them here. There are other drop in replacement compression tools. Pigz is of note because if you have a multicore system it can take advantage of those cores. – The Unix Janitor Jul 20 at 8:59
  • @DanielRosehill like gzip --rsyncable, pigz is rsync friendly. – The Unix Janitor Jul 20 at 9:07
  • The use case is that I'm backing up a couple of shared hosting servers (yes, there are still people that use them!). I have a fine (manually developed) methodology using a simple rsync pull (and mysqldump to bring in the databases - it's a WP site). The cPanel full backup however offers a more complete backup with cron jobs, emails and forwarders. Moreover, it is portable and easily restorable between hosts. It's downfall is that it's a full backup methodology. Pigz sounds great but I'm limited to what programs the host has installed. But I can run a test on my localhost. – Daniel Rosehill Jul 20 at 11:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.