Is there a utility like rsync, but with the speed of btrfs COW?

Question

rsync has various performance issues; I have never been able to make it perform a sync in O(#changes). Even if it uses modification time and not checksums, it still needs to generate a list of all changes files, which may take hours or run into some arbitrary limits. It is clearly possible however, for example with a copy-on-write filesystem, you can instantly transfer a 'minimal' diff without the minutes-long (or hours-long) overhead of rescanning the hard drive.

And certainly it would be possible with the following algorithm:

• my-ideal-rsync --modified-since "2020-10-10 12:00:00" would list/send/stream the modified files in optimal O(# modified files) time rather than O(# files on disk) time by recursively looking at each folder, and checking the folder's last-modified time if it's been modified since the time of the last transfer in the given command

• alternatively, my-ideal-rsync2 could do so without the above flag, by recursively comparing the folders of each system in lock-step:

  • recursively from the root, align all child inodes into pairs (source,dest):

    • if the source inode has the same last-modified time, don't recurse
    • if the source inode has a newer last-modified time, recurse if a directory (or send if a file)
    • if the source inode has an older last-modified time, throw an error
    • if the source or dest inode is missing, queue it for a possible mv operation, i.e. a queue of possible matches
  • (if a match is not found by the end of the recursion, then perform a deletion or creation, respectively)

Maybe there's a bug in the above algorithm, but it illustrates the concept. Is there anything like this?

Answer 1

Directory modification times are independent of file modification times. – roaima Jan 29 at 7:41

Due to my misunderstanding of directory modification times (at least on ext4, etc.), it seems such an algorithm is not possible, in the absence of using a filesystem that provides "a file in a subdirectory has been changed"-time. (Or, alternatively, setting the fs into read-only unless a file modification daemon is running to keep track of changes to later rsync... ew.)

Going to answer this tentatively as "such an algorithm is likely undoable with a normal filesystem because dir mtimes are independent of file mtimes; just use btrfs send

Answer 2

The rsync program does not record the difference, so it have to scan the entire directory tree.

In contrast, the git version control system tracks modifications made to files and directories and keeps all history, so that may be what you need.