276

I'm currently facing a problem on a linux box where as root I have commands returning error because inotify watch limit has been reached.

# tail -f /var/log/messages
[...]
tail: cannot watch '/var/log/messages': No space left on device
# inotifywatch -v /var/log/messages
Establishing watches...
Failed to watch /var/log/messages; upper limit on inotify watches reached!
Please increase the amount of inotify watches allowed per user via '/proc/sys/fs/inotify/max_user_watches'.`

I googled a bit and every solution I found is to increase the limit with:

sudo sysctl fs.inotify.max_user_watches=<some random high number>

But I was unable to find any information of the consequences of raising that value. I guess the default kernel value was set for a reason but it seems to be inadequate for particular usages. (e.g., when using Dropbox with a large number of folder, or software that monitors a lot of files)

So here are my questions:

  • Is it safe to raise that value and what would be the consequences of a too high value?
  • Is there a way to find out what are the currently set watches and which process set them to be able to determine if the reached limit is not caused by a faulty software?
Improve this question
1
  • You've probably checked this by now since this is 8 months old, but is your drive full? "tail: cannot watch '/var/log/messages': No space left on device"
    – froggythefrog
    Oct 2, 2019 at 18:33

2 Answers 2

Reset to default
384

Is it safe to raise that value and what would be the consequences of a too high value?

Yes, it's safe to raise that value and below are the possible costs [source]:

  • Each used inotify watch takes up 540 bytes (32-bit system), or 1 kB (double - on 64-bit) [sources: 1, 2]
  • This comes out of kernel memory, which is unswappable.
  • Assuming you set the max at 524288 and all were used (improbable), you'd be using approximately 256MB/512MB of 32-bit/64-bit kernel memory.
    • Note that your application will also use additional memory to keep track of the inotify handles, file/directory paths, etc. -- how much depends on its design.

To check the max number of inotify watches:

cat /proc/sys/fs/inotify/max_user_watches

To set max number of inotify watches

Temporarily:

  • Run sudo sysctl fs.inotify.max_user_watches= with your preferred value at the end.

Permanently (more detailed info):

  • put fs.inotify.max_user_watches=524288 into your sysctl settings. Depending on your system they might be in one of the following places:
    • Debian/RedHat: /etc/sysctl.conf
    • Arch: put a new file into /etc/sysctl.d/, e.g. /etc/sysctl.d/40-max-user-watches.conf
  • you may wish to reload the sysctl settings to avoid a reboot: sysctl -p (Debian/RedHat) or sysctl --system (Arch)

Check to see if the max number of inotify watches have been reached:

Use tail with the -f (follow) option on any old file, e.g. tail -f /var/log/dmesg: - If all is well, it will show the last 10 lines and pause; abort with Ctrl-C - If you are out of watches, it will fail with this somewhat cryptic error:

tail: cannot watch '/var/log/dmsg': No space left on device

To see what's using up inotify watches

find /proc/*/fd -lname anon_inode:inotify |
   cut -d/ -f3 |
   xargs -I '{}' -- ps --no-headers -o '%p %U %c' -p '{}' |
   uniq -c |
   sort -nr

The first column indicates the number of inotify fds (not the number of watches though) and the second shows the PID of that process [sources: 1, 2].

Improve this answer
11
  • 6
    I guess very few codes need the values higher than the default Dropbox may require a higher limit, depending on how many files you have. Iv'e raised mine with no issues. in fact, the dropbox notification (that occurs when it reaches its limit) explicitly tells you to raise it.
    – Falmarri
    May 25, 2011 at 20:55
  • Isn't there also a dnotify that works the same, but on directories? Maybe consolidate multiple files you are inotifying into a single directory?
    – LawrenceC
    Jun 16, 2011 at 13:36
  • 2
    @ultrasawblade- inotify replaced dnotify. dnotify was slow and buggy. inotify can be used on directories, and a directory will be "changed" when one of the files in that directory (one level deep) is modified. Directories are just files anyway.
    – beatgammit
    Jul 10, 2011 at 5:14
  • 7
    "Permanently: Replace the value within the /proc/sys/fs/inotify/max_user_watches" <-- this is incorrect. To make this permanent you need to change /etc/sysctl.conf
    – Merc
    May 7, 2016 at 4:38
  • 1
    @stackexchanger if you very sure that these are similar (e.g. you've tested), then please submit and edit to the Question. I myself haven't checked, and I know that distros tend to patch packages, so behavior could be different for the same packages. This is not to mention differences in package versions as well.
    – tshepang
    Jan 8, 2017 at 22:55
5

From this other topic how to get the number of inotify watches in use there is a reference to this script inotify-consumers which I found to be very helpful because it will show you exactly who is hogging the watchers.

Improve this answer
0

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .