rsync
has various performance issues; I have never been able to make it perform a sync in O(#changes)
. Even if it uses modification time and not checksums, it still needs to generate a list of all changes files, which may take hours or run into some arbitrary limits. It is clearly possible however, for example with a copy-on-write filesystem, you can instantly transfer a 'minimal' diff without the minutes-long (or hours-long) overhead of rescanning the hard drive.
And certainly it would be possible with the following algorithm:
my-ideal-rsync --modified-since "2020-10-10 12:00:00"
would list/send/stream the modified files in optimal O(# modified files) time rather than O(# files on disk) time by recursively looking at each folder, and checking the folder's last-modified time if it's been modified since the time of the last transfer in the given commandalternatively,
my-ideal-rsync2
could do so without the above flag, by recursively comparing the folders of each system in lock-step:recursively from the root, align all child inodes into pairs (source,dest):
- if the source inode has the same last-modified time, don't recurse
- if the source inode has a newer last-modified time, recurse if a directory (or send if a file)
- if the source inode has an older last-modified time, throw an error
- if the source or dest inode is missing, queue it for a possible
mv
operation, i.e. a queue of possible matches
- (if a match is not found by the end of the recursion, then perform a deletion or creation, respectively)
Maybe there's a bug in the above algorithm, but it illustrates the concept. Is there anything like this?
btrfs send
?