I ran into this problem, and none of these answers give you the answer of "how many watches is each process currently using?" The one-liners all give you how many instances are open, which is only part of the story, and the trace stuff is only useful to see new watches being opened.
TL;DR: This will get you a file with a list of open inotify instances and the number of watches they have, along with the pids and binaries that spawned them, sorted in descending order by watch count:
sudo lsof | awk '/anon_inode/ { gsub(/[urw]$/,"",$4); print "/proc/"$2"/fdinfo/"$4; }' | while read fdi; do count=$(sudo grep -c inotify $fdi); exe=$(sudo readlink $(dirname $(dirname $fdi))/exe); echo -e $count"\t"$fdi"\t"$exe; done | sort -nr > watches
That's a big ball of mess, so here's how I got there. To start, I ran a tail on a test file, and looked at the fd's it opened:
joel@gladstone:~$ tail -f test > /dev/null &
[3] 22734
joel@opx1:~$ ls -ahltr /proc/22734/fd
total 0
dr-xr-xr-x 9 joel joel 0 Feb 22 22:34 ..
dr-x------ 2 joel joel 0 Feb 22 22:34 .
lr-x------ 1 joel joel 64 Feb 22 22:35 4 -> anon_inode:inotify
lr-x------ 1 joel joel 64 Feb 22 22:35 3 -> /home/joel/test
lrwx------ 1 joel joel 64 Feb 22 22:35 2 -> /dev/pts/2
l-wx------ 1 joel joel 64 Feb 22 22:35 1 -> /dev/null
lrwx------ 1 joel joel 64 Feb 22 22:35 0 -> /dev/pts/2
So, 4 is the fd we want to investigate. Let's see what's in the fdinfo for that:
joel@opx1:~$ cat /proc/22734/fdinfo/4
pos: 0
flags: 00
mnt_id: 11
inotify wd:1 ino:15f51d sdev:ca00003 mask:c06 ignored_mask:0 fhandle-bytes:8 fhandle-type:1 f_handle:1df51500a75e538c
That looks like a entry for the watch at the bottom!
Let's try something with more watches, this time with the inotifywait utility, just watching whatever is in /tmp:
joel@gladstone:~$ inotifywait /tmp/* &
[4] 27862
joel@gladstone:~$ Setting up watches.
Watches established.
joel@gladstone:~$ ls -ahtlr /proc/27862/fd | grep inotify
lr-x------ 1 joel joel 64 Feb 22 22:41 3 -> anon_inode:inotify
joel@gladstone:~$ cat /proc/27862/fdinfo/3
pos: 0
flags: 00
mnt_id: 11
inotify wd:6 ino:7fdc sdev:ca00003 mask:fff ignored_mask:0 fhandle-bytes:8 fhandle-type:1 f_handle:dc7f0000551e9d88
inotify wd:5 ino:7fcb sdev:ca00003 mask:fff ignored_mask:0 fhandle-bytes:8 fhandle-type:1 f_handle:cb7f00005b1f9d88
inotify wd:4 ino:7fcc sdev:ca00003 mask:fff ignored_mask:0 fhandle-bytes:8 fhandle-type:1 f_handle:cc7f00006a1d9d88
inotify wd:3 ino:7fc6 sdev:ca00003 mask:fff ignored_mask:0 fhandle-bytes:8 fhandle-type:1 f_handle:c67f00005d1d9d88
inotify wd:2 ino:7fc7 sdev:ca00003 mask:fff ignored_mask:0 fhandle-bytes:8 fhandle-type:1 f_handle:c77f0000461d9d88
inotify wd:1 ino:7fd7 sdev:ca00003 mask:fff ignored_mask:0 fhandle-bytes:8 fhandle-type:1 f_handle:d77f00000053c98b
Aha! More entries! So we should have six things in /tmp then:
joel@opx1:~$ ls /tmp/ | wc -l
6
Excellent. My new inotifywait has one entry in its fd list (which is what the other one-liners here are counting), but six entries in its fdinfo file. So we can figure out how many watches a given fd for a given process is using by consulting its fdinfo file. Now to put it together with some of the above to grab a list of processes that have notify watches open and use that to count the entries in each fdinfo. This is similar to above, so I'll just dump the one-liner here:
sudo lsof | awk '/anon_inode/ { gsub(/[urw]$/,"",$4); print "/proc/"$2"/fdinfo/"$4; }' | while read fdi; do count=$(sudo grep -c inotify $fdi); echo -e $count"\t"$fdi; done
There's some thick stuff in here, but the basics are that I use awk to build an fdinfo path from the lsof output, grabbing the pid and fd number, stripping the u/r/w flag from the latter. Then for each constructed fdinfo path, I count the number of inotify lines and output the count and the pid.
It would be nice if I had what processes these pids represent in the same place though, right? I thought so. So, in a particularly messy bit, I settled on calling dirname twice on the fdinfo path to get pack to /proc/<pid>, adding /exe to it, and then running readlink on that to get the exe name of the process. Throw that in there as well, sort it by number of watches, and redirect it to a file for safe-keeping and we get:
sudo lsof | awk '/anon_inode/ { gsub(/[urw]$/,"",$4); print "/proc/"$2"/fdinfo/"$4; }' | while read fdi; do count=$(sudo grep -c inotify $fdi); exe=$(sudo readlink $(dirname $(dirname $fdi))/exe); echo -e $count"\t"$fdi"\t"$exe; done | sort -n > watches
Running that without sudo to just show my processes I launched above, I get:
joel@gladstone:~$ cat watches
6 /proc/4906/fdinfo/3 /usr/bin/inotifywait
1 /proc/22734/fdinfo/4 /usr/bin/tail
Perfect! A list of processes, fd's, and how many watches each is using, which is exactly what I needed.
