4

I work with a couple different nodejs live servers as part of my job and there seems to be some kind of leak within my tooling/workflow causing file watchers to accumulate over time until they hit the system limit. I then get the following cli errors:

Error from chokidar (<path-to-folder>): Error:
 ENOSPC: System limit for number of file watchers reached, watch '<path-to-folder>/<filename>'

I found the following command that should return the number of wile watchers of use:

find /proc/*/fd -user "$USER" -lname anon_inode:inotify -printf '%hinfo/%f\n' 2>/dev/null | xargs cat | grep -c '^inotify'

and it returns 515160, even though I've seemingly shut down all of my live servers. I have two sets of questions:

  1. How can I diagnose this? Can I get a list of all registered watchers, their watched path and corresponding PID, or something of that sort?
  2. Is there a way for me to kill them all? Is killing all file watchers even a good idea? Can I kill only watchers registered by my servers?

I'm running debian 11

3
  • 1
    You can see the counts per PID (and fd) if you remove the cat | part from the command. Commented Mar 29, 2022 at 13:21
  • 1
    I've had the same problem and I've just increased the inode/inotify (not sure which one right now) number and it ran fine, even though I only had like 4 node.js apps running. This behaviour is weird and I wonder what is causing it... Commented Mar 30, 2022 at 17:53
  • Do you have a source for that command? I'm interesting in learning how that works. Commented May 10 at 6:13

1 Answer 1

6
+25

The command you provided is searching in /proc for any file descriptors in /proc/*/fd/ which are symlinks to anon_inode:inotify. It's pretty straightforward to also report the commands and PIDs of these processes, along with the number of watches set:

#!/bin/bash

cd /proc
for p in [0-9]*
do cd $p
   if find fd -user "$USER" -lname anon_inode:inotify 2>/dev/null | grep -q .
   then IFS= read -d '' cmd < cmdline
        numwatch=$(cat fdinfo/* | grep -c '^inotify')
        [[ $numwatch -ge 1 ]] && printf '%s\n PID %s\t %s watches\n' "$cmd" "$p" "$numwatch"
   fi
   cd ..
done

In fact, I found that oligofren has already written a similar script, inotify-consumers, as an answer which has more nicely formatted output.

However, finding the actual paths being watched turns out to be more complicated. You only have the inodes from /proc/*/fdinfo, so you have to search through the whole filesystem to find the path which maps to the inode. This is an expensive operation.

There is a C++ program inotify-info which does this; also found from an answer here. I just built it on my machine and it works. Run with no arguments, it just lists numbers of watches per process, the same as the inotify-consumers script. Given a particular command name or PID, it also searches for the paths to the inodes watched by that process.

Killing all watchers is probably not a good idea, but after seeing which processes are using lots of watches you can make an informed choice.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .