7

I have a large file (2-3 GB, binary, undocumented format) that I use on two different computers (normally I use it on a desktop system but when I travel I put it on my laptop). I use rsync to transfer this file back and forth.

I make small updates to this file from time to time, changing less than 100 kB. This happens on both systems.

The problem with rsync as I understand it is that if it think a file has changed between source and destination it transfers the complete file. In my situation it feels like a big waste of time when just a small part of a file has changes. I envisage a protocol where the transfer agents on source and destination first checksums the whole file and then compare the result. When they realise that the checksum for the whole file is different, they split the file into two parts, A and B and checksum them separately.

Aha, B is identical on both machines, let's ignore that half. Now it splits A into A1 and A2. Ok, only A2 has changed. Split A2 into A2I and A2II and compare etc. Do this recursively until it has found e.g., three parts that are 1 MB each that differs between source and destination and then transfer just these parts and insert them in the right position at the destination file. Today with fast SSDs and multicore CPUs such parallelisation should be very efficient.

So my question is, are there any tools that works like this (or in another manner I couldn't imagine but with similar result) available today?

A request for clarification has been posted. I mostly use Mac so the filesystem is HFS+. Typically I start rsync like this

rsync -av --delete --progress --stats - in this cases I sometimes use SSH and sometimes rsyncd. When I use rsyncd I start it like this rsync --daemon --verbose --no-detach.

Second clarification: I ask for either a tool that just transfers the delta for a file that exists in two locations with small changes and/or if rsync really offers this. My experience with rsync is that it transfers the files in full (but now there is an answer that explains this: rsync needs an rsync server to be able to transfer just the deltas, otherwise (e.g., using ssh-shell) it transfers the whole file however much has changed).

closed as unclear what you're asking by psusi, jimmij, Networker, chaos, slm Jan 26 '15 at 2:49

Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • 1
    I'd say it's the best. – DisplayName Jan 25 '15 at 16:38
  • 1
    Rsync shouldn't retransmit the whole file if only a small part is changed. By default it transfers only the changes. Did you confirm that it indeed retransmits the whole file? How do you invoke rsync? – Marco Jan 25 '15 at 17:16
  • 1
    Typically something like this rsync -av --delete --progress --stats. Sometimes I use a rsync server rsync --daemon --verbose --no-detach. But the result is always the same. – d-b Jan 25 '15 at 19:40
  • 3
    The whole point of rsync is that it finds only the parts that have changed and sends only them. – psusi Jan 25 '15 at 20:22
  • @d-b is asking is there a tool that can efficiently re-copy a file. That is synchronise file a to be the same as file b. (May be this is what rsync already does, so may be this is the answer) – ctrl-alt-delor Jul 24 '16 at 13:06
8

Rsync will not use deltas but will transmit the full file in its entirety if it - as a single process - is responsible for the source and destination files. It can transmit deltas when there is a separate client and server process running on the source and destination machines.

The reason that rsync will not send deltas when it is the only process is that in order to determine whether it needs to send a delta it needs to read the source and destination files. By the time it's done that it might as well have just copied the file directly.

If you are using a command of this form you have only one rsync process:

rsync /path/to/local/file /network/path/to/remote/file

If you are using a command of this form you have two rsync processes (one on the local host and one on the remote) and deltas can be used:

rsync /path/to/local/file remote_host:/path/to/remote/file
  • 1
    Interesting. Didn't know that. As you can see in my comment here unix.stackexchange.com/questions/180992/… I sometimes start en rsync server on one of the machines (when I want to use checksums). I haven't noticed that it is faster (but again, so far I have only used rsync when I want very safe transfers and use checksums and then the checksumming takes a lot of time). – d-b Jan 25 '15 at 21:07
  • 1
    Hmm, after a second thought, if checksumming slows down the transfer, then again it can just transfer the whole file!? Any way to benchmark rsync transfers? – d-b Jan 25 '15 at 21:08
  • 1
    I don't see anywhere that shows examples of the local and remote path. Do you provide a hostname in one of the paths? If not, the deltas algorithm will not kick in. Also bear in mind that to calculate the deltas, the entire file has to be read (on both sides), which for large files will take a significant time. – roaima Jan 25 '15 at 21:23
  • 2
    yes, i use hostnames, either via ssh- or rsync-protocol (that is, not file://), such as rsync://remote.local:/myFile /myFile; – d-b Jan 25 '15 at 22:20
  • 1
    rsync is programmed to minimize network traffic, even if that means it has to do a lot of local disk IO. As far as rsync is concerned, network filesystems are als local, as rsync can open those files with a simple system call. – wurtel Jan 26 '15 at 11:51
2

From the DESCRIPTION section of man rsync:

Rsync is a fast and extraordinarily versatile file copying tool. It can copy locally, to/from another host over any remote shell, or to/from a remote rsync daemon. It offers a large number of options that control every aspect of its behavior and permit very flexible specification of the set of files to be copied. It is famous for its delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination.

So that would be a "no".

  • 1
    But how does it calculate the delta? In my experience, either if I transfer a directory with thousands of files, or this big file, it always transfers the whole file if there is a change. E.g., I sometimes also transfer directories with mp3s back and forth. If I change an ID-tag on such mp3 and get maybe 500 kB/s it takes 8-10 seconds to transfer the file although just a few bytes has changed. – d-b Jan 25 '15 at 19:37
  • 1
    @user23122 Are you using the -W switch? What switches do you use? And what filesystems? Edit this information into your question as it is all relevant. – jasonwryan Jan 25 '15 at 19:45
  • 1
    question updated with the requested information. – d-b Jan 25 '15 at 21:12
0

You can use RAID-1 (mirroring) for optimizimg this. This is getting strange with changed made to both sides but that makes using rsync strange, too. You should explain how this is to be handled.

  1. You can create a file (dd if=/dev/zero of=/path/to/syncfile.img bs=1M count=3500) which is a bit bigger than what your sync file will grow to in the near future.
  2. Then you put a loop device on top of that file (losetup /dev/loop5 /path/to/syncfile.img).
  3. You do this on both systems.
  4. On the system from where the changes shall be synced to the other one you make the block device of the other system available with network block devices (nbd).
  5. Create a RAID-1 array over both block devices: mdadm create /dev/md5 --raid-devices=2 --level=raid1 --bitmap=/path/to/ext3volume/sync-bitmap --assume-clean /dev/loop5 --write-mostly /dev/path/to/nbd. You must provide --bitmap=/path/to/ext3volume/sync-bitmap when you assemble the array later.
  6. Create a filesystem on the RAID mke2fs -j /dev/md5 and mount it somewhere.
  7. Copy the file to the volume. This should be done with a good network connection. Maybe there is a more clever way to write the file content directly into the block device which could be done locally but as the file content is mixed with filesystem metadata I don't know how to do that.

Now you can disconnect the network block device. This leads to a degraded RAID-1 on both sides. If you want to sync you have to: 1. unmount and top the RAID on the system you sync to 2. set up the nbd again 3. hot-add the nbd to the RAID on the sync source system.

Then both block devices will be synced. But due to the bitmap the source system knows without having to read on the other side which data has to be transferred.

problems

F*ck. Now that I have written all that I realize that this does not work (well) with changes on both sides (to different areas). It may work if you use --build instead of --create (which allows you to pretend that on both hosts the local block device is the master device).

Depending on how you want the changes in both directions handled you could try to make backups of the bitmap files (when both RAIDs are stopped!) and just run syncs in both directions. Or (if you want to write changes in one direction only), even worse, you run the sync, stop the RAID, replace the local bitmap with the remote one and sync again (and sync the bitmap files afterwards). That promises to be great fun.

LVM snapshots

Something similar can be done with LVM snapshots.

Not the answer you're looking for? Browse other questions tagged or ask your own question.