Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

If a set of files (several GBs big each) and each changes slightly every day (at random places, not only information appended at the end), how can it be copied efficiently? I mean, in the sense that only changed parts are updated, and not the whole files. That would mean the difference between copying some Kb here and there or some GBs.

share|improve this question
    
Binary diff? I think that's a thing but I don't think diff will do it – cat Feb 24 at 16:13
    
@cat: diff might be able to report differences, but here the problem is copying the small differences. – Quora Feans Feb 24 at 18:03
    
Well, I was thinking just copy the diffs -- rsync is better – cat Feb 24 at 18:06
    
In Server Fault: What program to use for incremental backup of large single file (2011-01-07) and in Stack Overflow: RSync single (archive) file that changes every time (2011-03-04). – Graham Perrin Mar 3 at 20:28
up vote 21 down vote accepted

The rsync program does exactly that. From the man page:

It is famous for its delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination. Rsync is widely used for backups and mirroring and as an improved copy command for everyday use.

share|improve this answer
    
In fact, rsync is actually running while writing this answer, backing up everything on this machine onto my file server (with zfs). – hlovdal Feb 24 at 16:53
    
For backup purposes, I'd like to recommend rdiff-backup (nongnu.org/rdiff-backup). I'm using it to backup four different machines, for several years now, with good results. – Thomas Padron-McCarthy Feb 25 at 8:17
    
Wikipedia has a good explanation of how rsync compares file chunks using rolling checksums. – Adam Katz Mar 7 at 19:12

You probably want a modern deduplicating backup program. Check out BorgBackup.

This will make multiple backups of each version of your large file, but will share the common content between the different versions, so the total space used for a given version of your large file will only be slightly more than the total disk space for a single version, assuming that the different versions only differ slightly.

share|improve this answer

If you're IO-limited in any way, use a filesystem such as BTRFS or ZFS that directly supports incremental backups without having to find the differences in files, such as what rsync has to do.

Using rsync is going to be slow and very IO-intensive.

Because if whatever application is writing changes to the files is in any way IO-limited, using rsync is going to take significant IO cycles away from the application that's the very reason the files exist. And if your backup process or system is IO-limited, rsync is going to take IO cycles away from your available backup bandwidth.

Just Google "rsync is slow". For example: rsync is very slow (factor 8 to 10) compared to cp on copying files from nfs-share to local dir

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.