python InotifyTree versus linux inotifywatch: which is more reliable?

Question

I’m working in ubuntu and need to monitor a directory tree for new subdirectories and files. These are populated by rsync; dozens of subdirectories and hundreds of files may be added in less than one second. I’m considering a few options:

option 1: Monitor the tree with python’s inotify.adapters.InotifyTree(). But the documentation contains this caveat: "IMPORTANT: Recursively monitoring paths is not a functionality provided by the kernel. Rather, we artificially implement it. As directory-created events are received, we create watches for the child directories on-the-fly. This means that there is potential for a race condition: if a directory is created and a file or directory is created inside before you (using the event_gen() loop) have a chance to observe it, then you are going to have a problem: If it is a file, then you will miss the events related to its creation, but, if it is a directory, then not only will you miss those creation events but this library will also miss them and not be able to add a watch for them."

option 2: Monitor the tree with the linux inotifywatch utility using the ‘--recursive’ option. That documentation includes this caveat: "Warning: If you use this option while watching the root directory of a large tree, it may take quite a while until all inotify watches are established, and events will not be received in this time."

Which of these two options do you think is less likely to miss new subdirectories and files? My sense is that option 2 is probably more reliable since inotifywatch is (probably) implemented in ‘C’. Thanks!

Answer 1

Let's start by properly understanding the problem. The warnings you read in both manuals come from "limitations and caveats" in man 7 inotify: https://man7.org/linux/man-pages/man7/inotify.7.html

This comes from the kernel limitation so it's important to know both pieces of software might have the same problems even if their respective manuals don't mention some.

My own guess here is that notifywatch's manual doesn't mention the race condition because it is unlikely to be relevant to most users. The manual states:

notifywatch listens for filesystem events using Linux's inotify(7) interface, then outputs a summary count of the events received on each file or directory.

If this software skips a few file create events in new subdirectories it will still be doing what it's supposed to: reporting stats.

---

What can you do?

I don't think either solution is guaranteed to be perfect as you seem to need and I would never let a race condition break my software if I knew it could be a problem. Don't just "hope" it'll all be fine!

Again from inotify 7 manual:

However, robust applications should allow for the fact that bugs in the monitoring logic or races of the kind described below may leave the cache inconsistent with the filesystem state. It is probably wise to do some consistency checking, and rebuild the cache when inconsistencies are detected.

The wise thing to do in your case would be to use an option similar to 1 or 2 and code around the race condition. When you detect new directories being created:

• Make a note of the new directory.
• wait until after you have read all available inotify events
• then scan the directory (pathlib.Path.iterdir()) to see if you somehow missed any files in the race condition.

How to wait?

Waiting for all available inotify events is tricky. Using option 1 (inotify from PyPi) you can fetch all available events using a short timeout:

gen = i.event_gen(yield_nones=False, timeout_s=0.1)

When you have new directories to scan, you can set a short timeout and scan those directories when it does timeout. When you have no new directories to scan you can use it without the timeout (never timeout).