8

If a set of files (several GBs big each) and each changes slightly every day (at random places, not only information appended at the end), how can it be copied efficiently? I mean, in the sense that only changed parts are updated, and not the whole files. That would mean the difference between copying some Kb here and there or some GBs.

25

The rsync program does exactly that. From the man page:

It is famous for its delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination. Rsync is widely used for backups and mirroring and as an improved copy command for everyday use.

  • In fact, rsync is actually running while writing this answer, backing up everything on this machine onto my file server (with zfs). – hlovdal Feb 24 '16 at 16:53
  • For backup purposes, I'd like to recommend rdiff-backup (nongnu.org/rdiff-backup). I'm using it to backup four different machines, for several years now, with good results. – Thomas Padron-McCarthy Feb 25 '16 at 8:17
  • Wikipedia has a good explanation of how rsync compares file chunks using rolling checksums. – Adam Katz Mar 7 '16 at 19:12
  • I tried rsync to "revert" back to an old VM hard disk image where only few files inside have been changed, sadly rsync seems to copy the whole file again (taking ~3 minutes writing ~300MB/s)? rsync -av --progress --partial --inplace arch-test1.qcow2.bak arch-test1.qcow2 – feedc0de Oct 20 '18 at 19:26
8

You probably want a modern deduplicating backup program. Check out BorgBackup.

This will make multiple backups of each version of your large file, but will share the common content between the different versions, so the total space used for a given version of your large file will only be slightly more than the total disk space for a single version, assuming that the different versions only differ slightly.

5

If you're IO-limited in any way, use a filesystem such as BTRFS or ZFS that directly supports incremental backups without having to find the differences in files, such as what rsync has to do.

Using rsync is going to be slow and very IO-intensive.

Because if whatever application is writing changes to the files is in any way IO-limited, using rsync is going to take significant IO cycles away from the application that's the very reason the files exist. And if your backup process or system is IO-limited, rsync is going to take IO cycles away from your available backup bandwidth.

Just Google "rsync is slow". For example: rsync is very slow (factor 8 to 10) compared to cp on copying files from nfs-share to local dir

  • This has the added benefit of preserving all metadata without having to check for it, and always knowing that your backup copy is as good as the original copy. rsync is fine for general use, but if you have a modern filesystem, it would be foolish to ignore its advantages. – user121391 Jan 11 '17 at 9:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.