How to tackle leaking file watchers on Debian 11?

Question

I work with a couple different nodejs live servers as part of my job and there seems to be some kind of leak within my tooling/workflow causing file watchers to accumulate over time until they hit the system limit. I then get the following cli errors:

Error from chokidar (<path-to-folder>): Error:
 ENOSPC: System limit for number of file watchers reached, watch '<path-to-folder>/<filename>'

I found the following command that should return the number of wile watchers of use:

find /proc/*/fd -user "$USER" -lname anon_inode:inotify -printf '%hinfo/%f\n' 2>/dev/null | xargs cat | grep -c '^inotify'

and it returns 515160, even though I've seemingly shut down all of my live servers. I have two sets of questions:

1. How can I diagnose this? Can I get a list of all registered watchers, their watched path and corresponding PID, or something of that sort?
2. Is there a way for me to kill them all? Is killing all file watchers even a good idea? Can I kill only watchers registered by my servers?

I'm running debian 11

Answer 1

The command you provided is searching in /proc for any file descriptors in /proc/*/fd/ which are symlinks to anon_inode:inotify. It's pretty straightforward to also report the commands and PIDs of these processes, along with the number of watches set:

#!/bin/bash

cd /proc
for p in [0-9]*
do cd $p
   if find fd -user "$USER" -lname anon_inode:inotify 2>/dev/null | grep -q .
   then IFS= read -d '' cmd < cmdline
        numwatch=$(cat fdinfo/* | grep -c '^inotify')
        [[ $numwatch -ge 1 ]] && printf '%s\n PID %s\t %s watches\n' "$cmd" "$p" "$numwatch"
   fi
   cd ..
done

In fact, I found that oligofren has already written a similar script, inotify-consumers, as an answer which has more nicely formatted output.

However, finding the actual paths being watched turns out to be more complicated. You only have the inodes from /proc/*/fdinfo, so you have to search through the whole filesystem to find the path which maps to the inode. This is an expensive operation.

There is a C++ program inotify-info which does this; also found from an answer here. I just built it on my machine and it works. Run with no arguments, it just lists numbers of watches per process, the same as the inotify-consumers script. Given a particular command name or PID, it also searches for the paths to the inodes watched by that process.

Killing all watchers is probably not a good idea, but after seeing which processes are using lots of watches you can make an informed choice.