1

I see a lot of useless recursive directory traversal in:

  • rsync
  • unison
  • owncloud

All above application (and many more) do recursive directory traversal over and over again. Since the application does not know if a file (deep inside the directory structure) has changed or not.

Inotify exists, but does AFAIK not scale to several thousand directories.

I guess there is no solution. But maybe I am missing something.

For example take this file:

a/b/c/d.txt

And two servers A and B.

If you have one million (or more) files and directories, all above applications need to check if the mtime of a file has changed.

Ain't there no faster way?

If the mtime change of a file would bobble up to the parent directories it would be enough to check the mtime of the upper directories on both servers. This way the application could skip the useless searching for changes.

The question: How to avoid recursive directory traversal for syncing two directories?

Please leave comments if you have questions or if you need more details. Thank you.

closed as too broad by Celada, Anthon, X Tian, G-Man, slm Jul 5 '15 at 18:48

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Avoid asking multiple distinct questions at once. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

6

There's a reason all those utilities use recursive directory traversal to discover changed files. There isn't any better way.

Inotify exists, but does AFAIK not scale to several thousand directories.

Not only that, but you have to listen continuously (say, as a daemon) and if you miss a single update then you have to recheck everything.

Ain't there no faster way?

No, sorry.

If you have a great idea, please do publish it (but a Q&A site isn't the place)!

  • If you find the answer then please publish it, But a Q&A site is not the place (Why?) – ctrl-alt-delor Jul 5 '15 at 16:01
  • Actually, no is the answer, as described, since rsync and other synchronization programs aren't aware of the file system status before they are called. So each time they are called, they have to learn about their whole world, like a new-born baby. – ikrabbe Jul 5 '15 at 16:18
  • @richard, what I meant to say is that if anyone has some great idea to improve the situation, perhaps an improvement to POSIX that's compelling enough to get adapted by several kernels, perhaps something nobody has though of before, that's fantastic, but it wouldn't be a question. – Celada Jul 6 '15 at 1:25
  • 1
    I love to ask question. Often I guess that there will be no answer. But to be sure I ask the question. A lot of my question get down-voted or closed since a lot of experts don't like question they can't answer. But for me this is no reason not to ask :-) – guettli Jul 7 '15 at 6:58

Not the answer you're looking for? Browse other questions tagged or ask your own question.