Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

After a recent upgrade to Fedora 15, I'm finding that a number of tools are failing with errors along the lines of:

tail: inotify resources exhausted
tail: inotify cannot be used, reverting to polling

It's not just tail that's reporting problems with inotify, either. Is there any way to interrogate the kernel to find out what process or processes are consuming the inotify resources? The current inotify-related sysctl settings look like this:

fs.inotify.max_user_instances = 128
fs.inotify.max_user_watches = 8192
fs.inotify.max_queued_events = 16384
share|improve this question
up vote 27 down vote accepted

It seems that if the process creates inotify instance via inotify_init(), the resulting file that represents filedescriptor in the /proc filesystem is a symlink to (non-existing) 'anon_inode:inotify' file.

$ cd /proc/5317/fd
$ ls -l
total 0
lrwx------ 1 puzel users 64 Jun 24 10:36 0 -> /dev/pts/25
lrwx------ 1 puzel users 64 Jun 24 10:36 1 -> /dev/pts/25
lrwx------ 1 puzel users 64 Jun 24 10:36 2 -> /dev/pts/25
lr-x------ 1 puzel users 64 Jun 24 10:36 3 -> anon_inode:inotify
lr-x------ 1 puzel users 64 Jun 24 10:36 4 -> anon_inode:inotify

Unless I misunderstood the concept, the following command should show you list of processes (their representation in /proc), sorted by number of inotify instances they use.

for foo in /proc/*/fd/*; do readlink -f $foo; done | grep inotify | sort | uniq -c | sort -nr
share|improve this answer
Excellent, thank you! I didn't know about the inotify inodes showing up in /proc. For my purposes, the command can be simplified to this: find /proc/*/fd/* -type l -lname 'anon_inode:inotify' -print – larsks Jun 25 '11 at 12:29
I'm glad it helped. And your solution with find -lname is indeed much nicer than mine with for loop and readlink. – Petr Uzel Jun 26 '11 at 10:44
Note that you could also be out of watches (not instances). E.g., on my system, that gives a low-teens number of instances, but there are many tens of thousands of watches from KDE's desktop search. Its too bad there isn't an easier way to check how many watches/instances are in use, since the kernel clearly knows... – derobert Jan 23 '13 at 16:18

You are probably running out of inotify watches rather than instances. To find out who's creating a lot of watches:

  1. Do echo 1 >> /sys/kernel/debug/tracing/events/syscalls/sys_exit_inotify_add_watch/enable to enable tracing of watch adds;
  2. Do cat /sys/kernel/debug/tracing/tracing_enabled to make sure it's set to 1 and if it isn't do echo 1 >> /sys/kernel/debug/tracing/tracing_enabled;
  3. Restart the processes with inotify instances (determined as described in Petr Uzel's answer) that you suspect of creating a lot of watches; and
  4. Read the file /sys/kernel/debug/tracing/trace to watch how many watches are created and by which processes.

When you're done, make sure to echo 0 into the enable file (and the tracing_enabled file if you had to enable that as well) to turn off tracing so you won't incur the performance hit of continuing to trace.

share|improve this answer
It was a backup application creating lots of inotify watches, and the solution in the accepted answer helped identify the culprit. However, I wasn't previously familiar with the system call tracing you've demonstrated here. Very cool. Thanks for the information! – larsks Jan 23 '13 at 16:35
are you sure it is '/sys/kernel/debug/tracing/tracing_enabled' ? On my system it seems the correct path is '/sys/kernel/debug/tracing/tracing_on'... – Kartoch Apr 10 '13 at 14:42
There is no /sys/kernel/debug/tracing/events/syscalls/sys_exit_inotify_add_watch/enable nor /sys/kernel/debug/tracing/tracing_enabled on Gentoo Linux, but /sys/kernel/debug/tracing/tracing_enabled exists. Why is that? – zeekvfu Dec 4 '13 at 16:32
find /proc/*/fd/* -type l -lname 'anon_inode:inotify' 2>/dev/null | cut -f 1-4 -d'/' |  sort | uniq -c  | sort -nr
share|improve this answer

To trace which processes consume inotify watches (not instances) you can use the dynamic ftrace feature of the kernel if it is enabled in your kernel.

The kernel option you need is CONFIG_DYNAMIC_FTRACE.

First mount the debugfs filesystem if it is not already mounted.

mount -t debugfs nodev /sys/kernel/debug

Go under the tracing subdirectory of this debugfs directory

cd /sys/kernel/debug/tracing

Enable tracing of function calls

echo function > current_tracer

Filter only SyS_inotify_add_watch system calls

echo SyS_inotify_add_watch > set_ftrace_filter

Clear the trace ring buffer if it wasn't empty

echo > trace

Enable tracing if it is not already enabled

echo 1 > tracing_on

Restart the suspected process (in my case it was crashplan, a backup application)

Watch the inotify_watch being exhausted

wc -l trace
cat trace


share|improve this answer

I have modified the script present in above to show the list of processes those are consuming inotify resources:

ps -p `find /proc/*/fd/* -type l -lname 'anon_inode:inotify' -print | sed s/'^\/proc\/'/''/ | sed s/'\/fd.*$'/''/`

I think there is a way to replace my double sed.

Yes. Use either

cut -f 3 -d '/'   


sed -e 's/^\/proc\/\([0-9]*\)\/.*/\1'  

and you'll only get the pid.
Also, if you add

2> /dev/null  

in the find, you'll get rid of any pesky error lines thrown by find. So this would work:

ps -p $(find /proc/*/fd/* -type l -lname 'anon_inode:inotify' -print 2> /dev/null | sed -e 's/^\/proc\/\([0-9]*\)\/.*/\1/')
share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.