I’m working in ubuntu and need to monitor a directory tree for new subdirectories and files. These are populated by rsync; dozens of subdirectories and hundreds of files may be added in less than one second. I’m considering a few options:

option 1: Monitor the tree with python’s inotify.adapters.InotifyTree(). But the documentation contains this caveat: "IMPORTANT: Recursively monitoring paths is not a functionality provided by the kernel. Rather, we artificially implement it. As directory-created events are received, we create watches for the child directories on-the-fly. This means that there is potential for a race condition: if a directory is created and a file or directory is created inside before you (using the event_gen() loop) have a chance to observe it, then you are going to have a problem: If it is a file, then you will miss the events related to its creation, but, if it is a directory, then not only will you miss those creation events but this library will also miss them and not be able to add a watch for them."

option 2: Monitor the tree with the linux inotifywatch utility using the ‘--recursive’ option. That documentation includes this caveat: "Warning: If you use this option while watching the root directory of a large tree, it may take quite a while until all inotify watches are established, and events will not be received in this time."

Which of these two options do you think is less likely to miss new subdirectories and files? My sense is that option 2 is probably more reliable since inotifywatch is (probably) implemented in ‘C’. Thanks!

  • It seems that the two warnings are equivalent, so the choice is the one that fits your development environment. OTOH the Python one puts code in between you and the filesystem's inotify functionality. More code, more bugs.
    – waltinator
    Mar 9 at 23:41
  • Silly question - why don't you just check rsync's output instead of hoping for the best with an independent tool that is reasonably likely to fail in unpredictable ways?
    – tink
    Mar 10 at 1:43
  • @tink - may application must quickly detect and process new files coming across the network from rsync
    – Tomasso
    Mar 10 at 23:08
  • Yes, but rsync's output it still authoritative as to what landed where .. and if you initiated the rsync from the receiving end picking that output for postprocessing should be quite doable.
    – tink
    Mar 10 at 23:17

1 Answer 1


Let's start by properly understanding the problem. The warnings you read in both manuals come from "limitations and caveats" in man 7 inotify: https://man7.org/linux/man-pages/man7/inotify.7.html

This comes from the kernel limitation so it's important to know both pieces of software might have the same problems even if their respective manuals don't mention some.

My own guess here is that notifywatch's manual doesn't mention the race condition because it is unlikely to be relevant to most users. The manual states:

notifywatch listens for filesystem events using Linux's inotify(7) interface, then outputs a summary count of the events received on each file or directory.

If this software skips a few file create events in new subdirectories it will still be doing what it's supposed to: reporting stats.

What can you do?

I don't think either solution is guaranteed to be perfect as you seem to need and I would never let a race condition break my software if I knew it could be a problem. Don't just "hope" it'll all be fine!

Again from inotify 7 manual:

However, robust applications should allow for the fact that bugs in the monitoring logic or races of the kind described below may leave the cache inconsistent with the filesystem state. It is probably wise to do some consistency checking, and rebuild the cache when inconsistencies are detected.

The wise thing to do in your case would be to use an option similar to 1 or 2 and code around the race condition. When you detect new directories being created:

  • Make a note of the new directory.
  • wait until after you have read all available inotify events
  • then scan the directory (pathlib.Path.iterdir()) to see if you somehow missed any files in the race condition.

How to wait?

Waiting for all available inotify events is tricky. Using option 1 (inotify from PyPi) you can fetch all available events using a short timeout:

gen = i.event_gen(yield_nones=False, timeout_s=0.1)

When you have new directories to scan, you can set a short timeout and scan those directories when it does timeout. When you have no new directories to scan you can use it without the timeout (never timeout).

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .