2

I'm a software developer mainly using Java on Arch Linux. I am experiencing a lot of resource failures recently, in my development environment and in the system itself.

When I start all the applications I usually use the system starts to throw errors, for instance:

In zsh:

mlenz@loki ~/[redacted] (git)-[master] % cd ..
VCS_INFO_detect_git:9: fork failed: resource temporarily unavailable

In Eclipse (my IDE):

Unhandled event loop exception
unable to create new native thread

In Wildfly (the Application Server we use for the current project) during startup:

13:19:54,962 ERROR [org.xnio.listener] (XNIO-1 I/O-2) XNIO001007: A channel event listener threw an exception: java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:714)
...

I suspected some ulimit problems, either the nproc or nofile limit, but I managed to set them to some unusually large values without the problem being fixed:

mlenz@loki ~ % ulimit -a               
-t: cpu time (seconds)              unlimited
-f: file size (blocks)              unlimited
-d: data seg size (kbytes)          unlimited
-s: stack size (kbytes)             8192
-c: core file size (blocks)         0
-m: resident set size (kbytes)      unlimited
-u: processes                       256106
-n: file descriptors                65535
-l: locked-in-memory size (kbytes)  1024
-v: address space (kbytes)          unlimited
-x: file locks                      unlimited
-i: pending signals                 63787
-q: bytes in POSIX msg queues       819200
-e: max nice                        0
-r: max rt priority                 0
-N 15:                              unlimited

I also found that the problem may be a system wide kernel resource limit, for instance the number of file handles, though /proc/sys/fs/file-nr says otherwise:

mlenz@loki ~ % cat /proc/sys/fs/file-nr 
13312   0   1629104

Our sysadmin checked lsof and thought that the number of lines there seemed extremely high, but I also checked to computer of another colleague and he had a similarly high number of lines there:

mlenz@loki ~ % lsof | wc -l
228787

My development environment is fairly unusual, in that I usually start 2 Wildfly application servers (one is a kind of dummy for an external system, while the other is used for our application). The problem seems to occur only when I start both application servers, along with some of the other applications I use on my computer. There are a lot of services running on my system, so I may have to increase some limit the system reaches, but I have absolutely no idea which one it might be.

Any ideas what I might be missing, or what I can check to see what's wrong?

  • How many processes are running? Do you have something spamming a lot of processes or threads? Increasing the limit doesn't help if you have a rogue process using up the resource as fast as it can. – Gilles Feb 16 '16 at 22:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.