4

I've been getting java.io.IOException: Too many open files while running a Kafka instance and using one topic with 1000 partitions so I started investigating the file descriptors limits in my ec2 vm. I cannot understand which is exactly the limit for open files on a Centos 7 machine since all the following commands produce different results. The commands are:

  • ulimit -a: open files 1024
  • lsof | wc -l: 298280
  • cat /proc/sys/fs/file-max: 758881 (which is consistent with /proc/sys/fs/file-nr)

If the actual limit is the one the last command produces then I am well below it (lsof | wc -l: 298280). But if this is the case, the output of the ulimit command is quite unclear to me since I am well above the 1024 open files.

According to the official documentation the best way to check for file descriptors in Centos is the /proc/sys/fs/file-max file but are there all these seemingly "inconsistencies" between these commands?

  • What do you get from ulimit -aH? – schily Oct 18 '18 at 15:15
  • I am getting 4096 open files. – Niko Oct 18 '18 at 15:18
  • So this is the hard limit that may even be raised if you are root. – schily Oct 18 '18 at 15:20
  • When I run cat /proc/sys/fs/file-nr I get 6112 0 758881. Which according to this indicates a total of 6112 allocated file descriptors (none of them are free - second number is zero). So if 4096 is the hard limit I am well above it and Kafka should have failed but it's still running. So how can this be the hard limit? And why the lsof output is that huge? – Niko Oct 18 '18 at 15:27
  • Did you try to raise you hard limit as root? – schily Oct 18 '18 at 15:30
6
  1. file-max is the maximum number of files that can be opened across the entire system. This is enforced at the kernel level.

  2. The man page for lsof states that:

In the absence of any options, lsof lists all open files belonging to all active processes.

This is consistent with your observations, since the number of files as reported by lsof is well below the file-max setting.

  1. Finally, ulimit is used to enforce resource limits at a user level. The parameter 'number of open files' is set at the user level, but is applied to each process started by that user. In this case, a single Kafka process can have up to 1024 file handles open (soft limit).

You can raise this limit on your own up to the hard limit, 4096. To raise the hard limit, root access is required.

If Kafka is running as a single process, you could find the number of files opened by that process by using lsof -p [PID].

Hope this clears things up.

  • This does clear things up a little bit. I wish I could upvote your answer but unfortunately I am lacking the reputation... Thanks! – Niko Oct 18 '18 at 15:40
  • 1
    @Niko, you should instead mark the answer as correct (set the check mark). – Juergen Oct 18 '18 at 15:49
0

That's a common mistake: to compare the results of a raw lsof call with supposed limit.

For the global limit (/proc/sys/fs/file-max) you should have a look at /proc/sys/fs/file-nr; the first value indicates what is used and the last value is the limit.

The OpenFile limit is for each process but it can be defined on a user; see command ulimit -Hn for user limits and see /etc/security/limits.conf for definitions. Generally applied with "app user" eg:"tomcat": set limit to 65000 to user tomcat that will apply on java process it runs.

If you want to check the limits applied on a process, get its PID and then:

cat /proc/${PID}/limits

If you want to check how many files are opened by a process, get its PID and then:

ls -1 /proc/${PID}/fd | wc -l (note for ls it's 'minus one', not to confuse with 'minus el')

If you want to know details with lsof but only for those file handles that count for the limit, try these:

lsof -p ${PID} | grep -P "^(\w+\s+){3}\d+\D+"

lsof -p ${PID} -d '^cwd,^err,^ltx,^mem,^mmap,^pd,^rtd,^txt' -a

Remark: the 'files' are files / pipe / tcp connections / etc.

Note that sometimes you'll probably need to be root or to use sudo to obtain correct result for the commands; without privilege, sometimes you don't see an error, you just get fewer results.

Finally, if you want to know what files on your filesystem are accessed by a process, have a look at:

lsof -p ${PID} | grep / | awk '{print $9}' | sort | uniq

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.