16

I've been successfully using rsync to synchronize my home directories between my laptop and my netbook (both using ext4 with Linux). My only problem with it is that every now and then I like to completely reorganize certain directories, and rsync sees this as deletions and creation of new files, which makes it very slow and inefficient.

Does anybody know of a good piece of software that handles file moves well?

I've found that unison should be capable of this, but it doesn't seem to work in practice. I tested unison by synchronizing two local directories, each with a large file in it, and it still detected my move as a deletion+creation and was in effect even slower than rsync.

Moreover, if I remember correctly from my looking at unison's source code a couple of months ago, it tried to do something like doing sha sums—and I don't want my ideal solution to do sha sums—that's too slow on large files/directories. I'd like something that notices things like:

" The last source counterpart of this file seems to have been deleted and there's a new file elsewhere in the source directory tree that has the same mdate, size, and inode number, so I'm just going to assume that this was a move and move its target counterpart accordingly instead of doing a delete+copy. "

It's important to me that I be able to sync these machines quickly.

Any suggestions?

7 Answers 7

6

You should take a look at rdiff-backup it does rsync underneath, but with the additional intelligence that you require (and it makes incremental backups capable of rolling back, but you can configure to switch that off).

The latest release is old (2009), but that is a sign of stability.

1
  • 3
    Anyone tried that?
    – mist
    Commented Aug 22, 2016 at 10:57
5

Unison does what you want, but only for remote synchronization. Try using ssh://localhost/path/to/dir as one of the roots.

Unison does base its decisions on file contents, it doesn't keep track of inode numbers.

4

You can handle moved and renamed files with rsync if the filesystems on the source and target directory have support for hard links. The idea is to let rsync reconstruct hard links before real transfer. You can find a good explanation here

We ended up with a simple solution that create an hidden tree of hard links inside the source/target directory, the basic script could be like this:

# Name of hidden directory
Shadow=".rsync_shadow"

# do real sync
rsync -ahHv --stats --no-inc-recursive --delete --delete-after "$Source"/ "$Target"

# update/create hidden dir of hard links in source
rsync -a --delete --link-dest="$Source" --exclude="/$Shadow" "$Source"/ "$Source/$Shadow"

# update/create hidden dir of hard links in target
rsync -a --delete --link-dest="$Target" --exclude="/$Shadow" "$Target"/ "$Target/$Shadow"

I have an example script on GitHub. But I advise you to do a large amount of testing before use this method on production.

3

I am using plain rsync for personal backups to a different device over ssh. Having found no satisfying and simple solution for my use case, I created my own script, rsync-prepare, in case anyone is interested.

It's "good enough" for me and it would be still quite some work to turn this into a robust package, so it will be a gist unless I or someone else finds the time and motivation to flesh it out.

Features:

  • it respects the rsync filters (uses rsync to get the file list of candidate files)
  • it works with remote hosts (unlike rsync-sidekick)
  • it's plain Python 3, which is available basically everywhere
  • it can detect circular moves/renames (a -> b, b -> c, c -> a)
  • it avoids computing hashsums if the filesize is unique in the set of files (then it cannot be a moved file)
1
  • 1
    As author of rsync-sidekick, this is one of the feature requests I've got multiple times — which I've not implemented! Glad to see a tool exists that handles all these use-cases. Commented Jan 30, 2023 at 16:13
2

Question is old, but relevant.

Here is the tool you're looking for: rsync-sidekick

This syncs 3 types of changes:

  1. Change in file modification timestamp (in case you change timestamps of photos en masse using exiftool)
  2. Rename of file/directory
  3. Moving a file from one directory to another

It's designed to run before rsync is run.

Disclaimer: I'm the author of above tool.

0

If you want to synchronize files between multiple machines, then you may want to simply use a centralized version control system like Subversion (or FSVS, which uses SVN as the backend). The nice part is that everything in your /home (that you add to version control) is then versioned and can be easily sync'd to other machines, or rolled back.

0

lsyncd also handles moved files with the rsync+ssh behaviour, although somewhat buggy (i get about 15% moves handled as delete/copy, yet to figure out why). Most of the time it does work.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .