I see a lot of useless recursive directory traversal in:
All above application (and many more) do recursive directory traversal over and over again. Since the application does not know if a file (deep inside the directory structure) has changed or not.
Inotify exists, but does AFAIK not scale to several thousand directories.
I guess there is no solution. But maybe I am missing something.
For example take this file:
And two servers A and B.
If you have one million (or more) files and directories, all above applications need to check if the mtime of a file has changed.
Ain't there no faster way?
If the mtime change of a file would bobble up to the parent directories it would be enough to check the mtime of the upper directories on both servers. This way the application could skip the useless searching for changes.
The question: How to avoid recursive directory traversal for syncing two directories?
Please leave comments if you have questions or if you need more details. Thank you.