What is exactly difference between INotify max_user_instances and max_user_watches?

Question

After reading some articles on the internet I am a little lost in understanding the difference between INotify max_user_instances and max_user_watches.

From official Linux man:

/proc/sys/fs/inotify/max_user_instances

This specifies an upper limit on the number of INotify instances that can be created per real user ID.

and

/proc/sys/fs/inotify/max_user_watches

This specifies an upper limit on the number of watches that can be created per real user ID.

Does it mean that max_user_instances is an instance of INotify process, which can monitor multiple filesystems and limit of that is specified by max_user_watches?

If the former is true, how does it work in practice? Each process, which has to monitor some filesystems is creating user instance of INotify (I think not really because it is related to user id)?

Currently, after deployment on Amazon Ec2 instance, I have an error like this:

 System.IO.IOException: The configured user limit (128) on the number of INotify instances has been reached.

If I understand correctly, there are too many instances created, which are monitoring for filesystem changes? What can be the cause of that?

Answer 1

An "instance" is single file descriptor, returned by inotify_init(). A single inotify file descriptor can be used by one process or shared by multiple processes, so they are rationed per-user instead of per-process.

A "watch" is a single file, observed by inotify instance. Each watch is unique, so they are also rationed per-user.

If an application creates too many instances, it either starts too many processes (and does not share inotify file descriptors between processes), or it is just plain buggy — for example, it may leak open inotify descriptors (open and then forget about them without closing).

There is also a possibility, that application is just poorly written, and uses multiple descriptors where one could suffice (you almost never need more than 1 inotify descriptor).

Open file descriptors can by listed via procfs:

ls -al /proc/<application process number>/fd/

A bit of extra information about a descriptor can seen in /proc/<PID>/fdinfo/<descriptor number>.