I'm a software developer mainly using Java on Arch Linux. I am experiencing a lot of resource failures recently, in my development environment and in the system itself.
When I start all the applications I usually use the system starts to throw errors, for instance:
mlenz@loki ~/[redacted] (git)-[master] % cd .. VCS_INFO_detect_git:9: fork failed: resource temporarily unavailable
In Eclipse (my IDE):
Unhandled event loop exception unable to create new native thread
In Wildfly (the Application Server we use for the current project) during startup:
13:19:54,962 ERROR [org.xnio.listener] (XNIO-1 I/O-2) XNIO001007: A channel event listener threw an exception: java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Thread.java:714) ...
I suspected some
ulimit problems, either the
nofile limit, but I managed to set them to some unusually large values without the problem being fixed:
mlenz@loki ~ % ulimit -a -t: cpu time (seconds) unlimited -f: file size (blocks) unlimited -d: data seg size (kbytes) unlimited -s: stack size (kbytes) 8192 -c: core file size (blocks) 0 -m: resident set size (kbytes) unlimited -u: processes 256106 -n: file descriptors 65535 -l: locked-in-memory size (kbytes) 1024 -v: address space (kbytes) unlimited -x: file locks unlimited -i: pending signals 63787 -q: bytes in POSIX msg queues 819200 -e: max nice 0 -r: max rt priority 0 -N 15: unlimited
I also found that the problem may be a system wide kernel resource limit, for instance the number of file handles, though
/proc/sys/fs/file-nr says otherwise:
mlenz@loki ~ % cat /proc/sys/fs/file-nr 13312 0 1629104
Our sysadmin checked
lsof and thought that the number of lines there seemed extremely high, but I also checked to computer of another colleague and he had a similarly high number of lines there:
mlenz@loki ~ % lsof | wc -l 228787
My development environment is fairly unusual, in that I usually start 2 Wildfly application servers (one is a kind of dummy for an external system, while the other is used for our application). The problem seems to occur only when I start both application servers, along with some of the other applications I use on my computer. There are a lot of services running on my system, so I may have to increase some limit the system reaches, but I have absolutely no idea which one it might be.
Any ideas what I might be missing, or what I can check to see what's wrong?