10

I've been successfully using rsync to synchronize my home directories between my laptop and my netbook (both using ext4 with Linux). My only problem with it is that every now and then I like to completely reorganize certain directories, and rsync sees this as deletions and creation of new files, which makes it very slow and inefficient.

Does anybody know of a good piece of software that handles file moves well?

I've found that unison should be capable of this, but it doesn't seem to work in practice. I tested unison by synchronizing two local directories, each with a large file in it, and it still detected my move as a deletion+creation and was in effect even slower than rsync.

Moreover, if I remember correctly from my looking at unison's source code a couple of months ago, it tried to do something like doing sha sums—and I don't want my ideal solution to do sha sums—that's too slow on large files/directories. I'd like something that notices things like:

" The last source counterpart of this file seems to have been deleted and there's a new file elsewhere in the source directory tree that has the same mdate, size, and inode number, so I'm just going to assume that this was a move and move its target counterpart accordingly instead of doing a delete+copy. "

It's important to me that I be able to sync these machines quickly.

Any suggestions?

6

You should take a look at rdiff-backup it does rsync underneath, but with the additional intelligence that you require (and it makes incremental backups capable of rolling back, but you can configure to switch that off).

The latest release is old (2009), but that is a sign of stability.

  • 2
    Anyone tried that? – mist Aug 22 '16 at 10:57
3

Unison does what you want, but only for remote synchronization. Try using ssh://localhost/path/to/dir as one of the roots.

Unison does base its decisions on file contents, it doesn't keep track of inode numbers.

1

You can handle moved and renamed files with rsync if the filesystems on the source and target directory have support for hard links. The idea is to let rsync reconstruct hard links before real transfer. You can find a good explanation here

We ended up with a simple solution that create an hidden tree of hard links inside the source/target directory, the basic script could be like this:

# Name of hidden directory
Shadow=".rsync_shadow"

# do real sync
rsync -ahHv --stats --no-inc-recursive --delete --delete-after "$Source"/ "$Target"

# update/create hidden dir of hard links in source
rsync -a --delete --link-dest="$Source" --exclude="/$Shadow" "$Source"/ "$Source/$Shadow"

# update/create hidden dir of hard links in target
rsync -a --delete --link-dest="$Target" --exclude="/$Shadow" "$Target"/ "$Target/$Shadow"

I have an example script on GitHub. But I advise you to do a large amount of testing before use this method on production.

0

If you want to synchronize files between multiple machines, then you may want to simply use a centralized version control system like Subversion (or FSVS, which uses SVN as the backend). The nice part is that everything in your /home (that you add to version control) is then versioned and can be easily sync'd to other machines, or rolled back.

0

lsyncd also handles moved files with the rsync+ssh behaviour, although somewhat buggy (i get about 15% moves handled as delete/copy, yet to figure out why). Most of the time it does work.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.