If a set of files (several GBs big each) and each changes slightly every day (at random places, not only information appended at the end), how can it be copied efficiently? I mean, in the sense that only changed parts are updated, and not the whole files. That would mean the difference between copying some Kb here and there or some GBs.
asked Feb 24, 2016 at 15:58
3 Answers
The rsync program does exactly that. From the man page:
It is famous for its delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination. Rsync is widely used for backups and mirroring and as an improved copy command for everyday use.
-
In fact, rsync is actually running while writing this answer, backing up everything on this machine onto my file server (with zfs).– hlovdalFeb 24, 2016 at 16:53
-
For backup purposes, I'd like to recommend rdiff-backup (nongnu.org/rdiff-backup). I'm using it to backup four different machines, for several years now, with good results. Feb 25, 2016 at 8:17
-
Wikipedia has a good explanation of how
rsynccompares file chunks using rolling checksums. Mar 7, 2016 at 19:12 -
I tried rsync to "revert" back to an old VM hard disk image where only few files inside have been changed, sadly rsync seems to copy the whole file again (taking ~3 minutes writing ~300MB/s)?
rsync -av --progress --partial --inplace arch-test1.qcow2.bak arch-test1.qcow2– feedc0deOct 20, 2018 at 19:26
You probably want a modern deduplicating backup program. Check out BorgBackup.
This will make multiple backups of each version of your large file, but will share the common content between the different versions, so the total space used for a given version of your large file will only be slightly more than the total disk space for a single version, assuming that the different versions only differ slightly.
answered Feb 25, 2016 at 0:00
If you're IO-limited in any way, use a filesystem such as BTRFS or ZFS that directly supports incremental backups without having to find the differences in files, such as what rsync has to do.
Using rsync is going to be slow and very IO-intensive.
Because if whatever application is writing changes to the files is in any way IO-limited, using rsync is going to take significant IO cycles away from the application that's the very reason the files exist. And if your backup process or system is IO-limited, rsync is going to take IO cycles away from your available backup bandwidth.
Just Google "rsync is slow". For example: rsync is very slow (factor 8 to 10) compared to cp on copying files from nfs-share to local dir
-
This has the added benefit of preserving all metadata without having to check for it, and always knowing that your backup copy is as good as the original copy. rsync is fine for general use, but if you have a modern filesystem, it would be foolish to ignore its advantages. Jan 11, 2017 at 9:04
diffwill do itdiffmight be able to report differences, but here the problem is copying the small differences.rsyncis better