1

What's the optimal, smart parameters to rsync to achieve fastest checking of huge folders that rarely change and have lots of small files. I'm using the following command to sync a local disk to a USB3 disk at the end of each day (having done the opposite at the start of that day) and it still takes a minute even if there were no changes -- can one do the checks faster? For example, I thought each folder has a modification time, so no need recursively go into them if /Source/A and /Target/A have the same modified time? Or is it dangerous and rsync must check each of the nested thousand little files? Other ideas? Thanks

rsync -a --delete --stats /Source/ /Target/

Number of files: 208,645 (reg: 187,331, dir: 16,910, link: 4,404)
Number of created files: 0
Number of deleted files: 0
Number of regular files transferred: 0
Total file size: 11,103,192,078 bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 65,536
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 4,126,114
Total bytes received: 17,677

sent 4,126,114 bytes  received 17,677 bytes  236,788.06 bytes/sec
total size is 11,103,192,078  speedup is 2,679.48
Improve this question
3
  • 1
    By default rsync do the best. mankier.com/1/rsync#-c serverfault.com/questions/211005/…
    – gapsf
    Sep 24, 2022 at 12:08
  • 1
    folders only get modified time change if a file is create or removed within it, so every file's times need to be checked too. you could use a tool like unison so the changes are written continuously to the usb disk, or you could use something like git, and use the usb driive as a repo to be pushed and pull to.
    – toppk
    Sep 24, 2022 at 15:27
  • 1
    @roaima my source is a fixed NTFS and my portable USB3 backup stick carries ext4 at the moment...
    – rych
    Sep 25, 2022 at 13:46

1 Answer 1

Reset to default
2

You'll not get much faster than that with rsync between two local devices, particularly where one of them is non-native NTFS. It's already trying to skip files that "look like" they are unchanged.

You might get a better throughput if you were transferring files between native filesystems (eg ext4) or two different systems, but probably not a lot.

There are no directory-based optimisations available, as every entry needs to be scanned to check if it's a directory, and modification times don't get propagated to parents.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .