Difference between ulimit, lsof, cat /proc/sys/fs/file-max

Question

I've been getting java.io.IOException: Too many open files while running a Kafka instance and using one topic with 1000 partitions so I started investigating the file descriptors limits in my ec2 vm. I cannot understand which is exactly the limit for open files on a Centos 7 machine since all the following commands produce different results. The commands are:

• ulimit -a: open files 1024
• lsof | wc -l: 298280
• cat /proc/sys/fs/file-max: 758881 (which is consistent with /proc/sys/fs/file-nr)

If the actual limit is the one the last command produces then I am well below it (lsof | wc -l: 298280). But if this is the case, the output of the ulimit command is quite unclear to me since I am well above the 1024 open files.

According to the official documentation the best way to check for file descriptors in Centos is the /proc/sys/fs/file-max file but are there all these seemingly "inconsistencies" between these commands?

Answer 1

1. file-max is the maximum number of files that can be opened across the entire system. This is enforced at the kernel level.

2. The man page for lsof states that:

In the absence of any options, lsof lists all open files belonging to all active processes.

This is consistent with your observations, since the number of files as reported by lsof is well below the file-max setting.

3. Finally, ulimit is used to enforce resource limits at a user level. The parameter 'number of open files' is set at the user level, but is applied to each process started by that user. In this case, a single Kafka process can have up to 1024 file handles open (soft limit).

You can raise this limit on your own up to the hard limit, 4096. To raise the hard limit, root access is required.

If Kafka is running as a single process, you could find the number of files opened by that process by using lsof -p [PID].

Hope this clears things up.

Answer 2

That's a common mistake: to compare the results of a raw lsof call with supposed limit.

For the global limit (/proc/sys/fs/file-max) you should have a look at /proc/sys/fs/file-nr; the first value indicates what is used and the last value is the limit.

The OpenFile limit is for each process but it can be defined on a user; see command ulimit -Hn for user limits and see /etc/security/limits.conf for definitions. Generally applied with "app user" eg:"tomcat": set limit to 65000 to user tomcat that will apply on java process it runs.

If you want to check the limits applied on a process, get its PID and then:

cat /proc/${PID}/limits

If you want to check how many files are opened by a process, get its PID and then:

ls -1 /proc/${PID}/fd | wc -l (note for ls it's 'minus one', not to confuse with 'minus el')

If you want to know details with lsof but only for those file handles that count for the limit, try these:

lsof -p ${PID} | grep -P "^(\w+\s+){3}\d+\D+"

lsof -p ${PID} -d '^cwd,^err,^ltx,^mem,^mmap,^pd,^rtd,^txt' -a

Remark: the 'files' are files / pipe / tcp connections / etc.

Note that sometimes you'll probably need to be root or to use sudo to obtain correct result for the commands; without privilege, sometimes you don't see an error, you just get fewer results.

Finally, if you want to know what files on your filesystem are accessed by a process, have a look at:

lsof -p ${PID} | grep / | awk '{print $9}' | sort | uniq