Limit for open files

Question

I have a java application that is failing with

java.io.IOException: Too many open files
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) ~[na:1.8.0-internal]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:477) ~[na:1.8.0-internal]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:287) ~[na:1.8.0-internal]
        at org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:455) ~[tomcat-embed-core-8.5.27.jar!/:8.5.27]
        at java.lang.Thread.run(Thread.java:785) [na:1.8.0-internal]

my limit for open files is 30K

$ ulimit -a
...
open files                      (-n) 30480
...

I'm wondering now, what is the proper check...

lsof | grep 123 | wc -l (123 is pid of java application) returns number 45633.

How it could be over 30K?

On the other hand lsof -p 123| wc -l returns only 771 - it's not over the limit.

Can someone help me to understand what's going on here? What am I missing? Is the limit sum for all user processes (that's what I'd expect)?

This is RedHat 7, Lsof revision 4.87.

edit: ok, I believe I know what is the discrepancy - lsof -p ... shows open files for parent only while there are subprocesses

Thanks to a comment from @schily I found that limit per subprocess is 4K only (and not 30K).

Answer 1

Solution in my case, as I'm running it as a service (I didn't mention as I didn't know it's important), was to add

LimitNOFILE=8192

to my service file under /etc/systemd

That service file specifies, that it has to be run as some specific user, but still the limits were not used for this user.