I understand that rsync is very versatile, so I'm just wondering what the best way to use it would be in my situation ... which is:

~700GB project (in a single folder project/ with various sub-directories) on a workstation which I SSH into to do processing and analysis. This work station mounts an NFS share which I don't have direct access to. Is it possible to use the delta functionality of rsync to make incremental backups of the project?

I am already using rsnapshot to backup but I am hoping there is a simple command with rsync that will allow me to delta copy the project over. It changes very little, so the delta copy would really cut down on bandwidth.

I have tried

$ rsync --no-whole-file -rv project/ /nfs/destination/project/

But this did not work... I have a feeling I am not using it correctly.

  • When you rsync files on /project and cd into /project. You can run : rsync --no-whole-file -rv * /nfs/destination/project/ that will include subdirectories on /project. – supriady Jan 5 '17 at 4:02

The delta functionality of rsync only works between two different hosts.

It's a trade-off between network speed and file IO, which results in the two hosts both reading the entire file as well as copying and writing the parts that are different. Given a reasonable assumption that a read and write is faster than two reads and some writing, it doesn't make sense to employ the delta algorithm between two copies of a file on the same host.

You could theoretically try to dupe rsync into thinking it was running across a network with a command like this:

rsync -rv project/ localhost:/nfs/destination/project/

but you would find that the host ended up reading each file under project/ as well as the target file under /nfs/destination/project/ to determine the differences. At this point you might as well have copied a file in its entirety.

The only shortcut that avoids a full file write is when you have identical file times and file size on the source and destination versions.

One option is to see whether the administrator of your NFS server will give you an rsync service to which you can connect. (rsyncd is one option on a Linux/Unix platform, or DeltaCopy Server on a Windows platform. Both can be configured to require user authentication.) This would allow you to access rsync's delta algorithm and write only changes.

Another option is to see whether your storage admins have a better suggestion for backups, such as block-based change tracking. It might be that they could create a backup schedule that worked for you as a user.

  • So if I use the -t flag when I copy, this will work? File sizes shouldn't be changing, except possibly any files that have changed which I want to copy anyway... – reas0n Jan 5 '17 at 15:35
  • 1
    @Elijah personally I tend to use -aSH and then negate any characteristics I don't want (like --no-perms or --no-group). Since -a corresponds to -rlptgoD and you're only using -r, you could use -rt, yes. – roaima Jan 5 '17 at 17:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.