Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

After a recent upgrade to Fedora 15, I'm finding that a number of tools are failing with errors along the lines of:

tail: inotify resources exhausted
tail: inotify cannot be used, reverting to polling

It's not just tail that's reporting problems with inotify, either. Is there any way to interrogate the kernel to find out what process or processes are consuming the inotify resources? The current inotify-related sysctl settings look like this:

fs.inotify.max_user_instances = 128
fs.inotify.max_user_watches = 8192
fs.inotify.max_queued_events = 16384
share|improve this question
add comment

4 Answers

up vote 18 down vote accepted

It seems that if the process creates inotify instance via inotify_init(), the resulting file that represents filedescriptor in the /proc filesystem is a symlink to (non-existing) 'anon_inode:inotify' file.

$ cd /proc/5317/fd
$ ls -l
total 0
lrwx------ 1 puzel users 64 Jun 24 10:36 0 -> /dev/pts/25
lrwx------ 1 puzel users 64 Jun 24 10:36 1 -> /dev/pts/25
lrwx------ 1 puzel users 64 Jun 24 10:36 2 -> /dev/pts/25
lr-x------ 1 puzel users 64 Jun 24 10:36 3 -> anon_inode:inotify
lr-x------ 1 puzel users 64 Jun 24 10:36 4 -> anon_inode:inotify

Unless I misunderstood the concept, the following command should show you list of processes (their representation in /proc), sorted by number of inotify instances they use.

for foo in /proc/*/fd/*; do readlink -f $foo; done | grep inotify | sort | uniq -c | sort -nr
share|improve this answer
4  
Excellent, thank you! I didn't know about the inotify inodes showing up in /proc. For my purposes, the command can be simplified to this: find /proc/*/fd/* -type l -lname 'anon_inode:inotify' -print –  larsks Jun 25 '11 at 12:29
    
I'm glad it helped. And your solution with find -lname is indeed much nicer than mine with for loop and readlink. –  Petr Uzel Jun 26 '11 at 10:44
1  
Note that you could also be out of watches (not instances). E.g., on my system, that gives a low-teens number of instances, but there are many tens of thousands of watches from KDE's desktop search. Its too bad there isn't an easier way to check how many watches/instances are in use, since the kernel clearly knows... –  derobert Jan 23 '13 at 16:18
add comment

You are probably running out of inotify watches rather than instances. To find out who's creating a lot of watches:

  1. Do echo 1 >> /sys/kernel/debug/tracing/events/syscalls/sys_exit_inotify_add_watch/enable to enable tracing of watch adds;
  2. Do cat /sys/kernel/debug/tracing/tracing_enabled to make sure it's set to 1 and if it isn't do echo 1 >> /sys/kernel/debug/tracing/tracing_enabled;
  3. Restart the processes with inotify instances (determined as described in Petr Uzel's answer) that you suspect of creating a lot of watches; and
  4. Read the file /sys/kernel/debug/tracing/trace to watch how many watches are created and by which processes.

When you're done, make sure to echo 0 into the enable file (and the tracing_enabled file if you had to enable that as well) to turn off tracing so you won't incur the performance hit of continuing to trace.

share|improve this answer
    
It was a backup application creating lots of inotify watches, and the solution in the accepted answer helped identify the culprit. However, I wasn't previously familiar with the system call tracing you've demonstrated here. Very cool. Thanks for the information! –  larsks Jan 23 '13 at 16:35
1  
are you sure it is '/sys/kernel/debug/tracing/tracing_enabled' ? On my system it seems the correct path is '/sys/kernel/debug/tracing/tracing_on'... –  Kartoch Apr 10 '13 at 14:42
    
There is no /sys/kernel/debug/tracing/events/syscalls/sys_exit_inotify_add_watch/enable nor /sys/kernel/debug/tracing/tracing_enabled on Gentoo Linux, but /sys/kernel/debug/tracing/tracing_enabled exists. Why is that? –  zeekvfu Dec 4 '13 at 16:32
add comment
find /proc/*/fd/* -type l -lname 'anon_inode:inotify' 2>/dev/null | cut -f 1-4 -d'/' |  sort | uniq -c  | sort -nr
share|improve this answer
add comment

I have modified the script present in above to show the list of processes those are consuming inotify resources:

ps -p `find /proc/*/fd/* -type l -lname 'anon_inode:inotify' -print | sed s/'^\/proc\/'/''/ | sed s/'\/fd.*$'/''/`

I think there is a way to replace my double sed.


Yes. Use either

cut -f 3 -d '/'   

or

sed -e 's/^\/proc\/\([0-9]*\)\/.*/\1'  

and you'll only get the pid.
Also, if you add

2> /dev/null  

in the find, you'll get rid of any pesky error lines thrown by find. So this would work:

ps -p $(find /proc/*/fd/* -type l -lname 'anon_inode:inotify' -print 2> /dev/null | sed -e 's/^\/proc\/\([0-9]*\)\/.*/\1/')
share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.