8

Multiple sessions of the same user. When one of them gets to the point that it can no longer run new programs, none of them can, not even a new login of that user. Other users can still run new programs just fine, including new logins.

Normally user limits are in limits.conf, but its documentation says "please note that all limit settings are set per login. They are not global, nor are they permanent; existing only for the duration of the session."

I'm nowhere close to running out of ram (44GB available), but I can't figure out what else to look at. What limits exist that would have a global effect on all sessions using the same UID, but not other UIDs?

Edited on 6/12/16 at 8:45p to add:

While writing the below I realized that the problem could be X11 related. This user account on this box is used nearly exclusively for GUI applications. Is there a good text based program I can try to run from bash that will use lots of resources and give good error messages? The box does not get to the point where it cannot even run ls.

Unfortunately, the GUI programs this problem normally affects (Chrome and Firefox) do not do a good job of leaving error messages behind. Chrome tabs will start showing up blank or with the completely useless "Aw, Snap!" error. Firefox simply will refuse to start. The only even partially helpful error messages I managed to obtain came from trying to start Firefox from bash:

[pascal@firefox ~]$ firefox --display=:0 --safe-mode
Assertion failure: ((bool)(__builtin_expect(!!(!NS_FAILED_impl(rv)), 1))) && thread (Should successfully create image decoding threads), at /builddir/build/BUILD/firefox-45.2.0/firefox-45.2.0esr/image/DecodePool.cpp:359
#01: ???[/usr/lib64/firefox/libxul.so +0x10f2165]
#02: ???[/usr/lib64/firefox/libxul.so +0xa2dd2c]
#03: ???[/usr/lib64/firefox/libxul.so +0xa2ee29]
#04: ???[/usr/lib64/firefox/libxul.so +0xa2f4c1]
#05: ???[/usr/lib64/firefox/libxul.so +0xa3095d]
#06: ???[/usr/lib64/firefox/libxul.so +0xa52d44]
#07: ???[/usr/lib64/firefox/libxul.so +0xa4c051]
#08: ???[/usr/lib64/firefox/libxul.so +0x1096257]
#09: ???[/usr/lib64/firefox/libxul.so +0x1096342]
#10: ???[/usr/lib64/firefox/libxul.so +0x1dba68f]
#11: ???[/usr/lib64/firefox/libxul.so +0x1dba805]
#12: ???[/usr/lib64/firefox/libxul.so +0x1dba8b9]
#13: ???[/usr/lib64/firefox/libxul.so +0x1e3e6be]
#14: ???[/usr/lib64/firefox/libxul.so +0x1e48d1f]
#15: ???[/usr/lib64/firefox/libxul.so +0x1e48ddd]
#16: ???[/usr/lib64/firefox/libxul.so +0x20bf7bc]
#17: ???[/usr/lib64/firefox/libxul.so +0x20bfae6]
#18: ???[/usr/lib64/firefox/libxul.so +0x20bfe5b]
#19: ???[/usr/lib64/firefox/libxul.so +0x21087cd]
#20: ???[/usr/lib64/firefox/libxul.so +0x2108cd2]
#21: ???[/usr/lib64/firefox/libxul.so +0x210aef4]
#22: ???[/usr/lib64/firefox/libxul.so +0x22578b1]
#23: ???[/usr/lib64/firefox/libxul.so +0x228ba43]
#24: ???[/usr/lib64/firefox/libxul.so +0x228be1d]
#25: XRE_main[/usr/lib64/firefox/libxul.so +0x228c073]
#26: ???[/usr/lib64/firefox/firefox +0x4c1d]
#27: ???[/usr/lib64/firefox/firefox +0x436d]
#28: __libc_start_main[/lib64/libc.so.6 +0x21b15]
#29: ???[/usr/lib64/firefox/firefox +0x449d]
#30: ??? (???:???)
Segmentation fault
[pascal@firefox ~]$ firefox --display=:0 --safe-mode -g
1465632860286DeferredSave.extensions.jsonWARNWrite failed: Error: Could not create new thread! (resource://gre/modules/PromiseWorker.jsm:173:18) JS Stack trace: [email protected]:173:18 < [email protected]:292:9 < [email protected]:315:40 < [email protected]:933:23 < [email protected]:812:7 < this.PromiseWalker.scheduleWalkerLoop/<@Promise-backend.js:746:1 < [email protected]:770:1 < [email protected]:284:9
1465632860287addons.xpi-utilsWARNFailed to save XPI database: Error: Could not create new thread! (resource://gre/modules/PromiseWorker.jsm:173:18) JS Stack trace: [email protected]:173:18 < [email protected]:292:9 < [email protected]:315:40 < [email protected]:933:23 < [email protected]:812:7 < this.PromiseWalker.scheduleWalkerLoop/<@Promise-backend.js:746:1 < [email protected]:770:1 < [email protected]:284:9
1465632860288addons.xpi-utilsWARNFailed to save XPI database: Error: Could not create new thread! (resource://gre/modules/PromiseWorker.jsm:173:18) JS Stack trace: [email protected]:173:18 < [email protected]:292:9 < [email protected]:315:40 < [email protected]:933:23 < [email protected]:812:7 < this.PromiseWalker.scheduleWalkerLoop/<@Promise-backend.js:746:1 < [email protected]:770:1 < [email protected]:284:9
1465632860289addons.xpi-utilsWARNFailed to save XPI database: Error: Could not create new thread! (resource://gre/modules/PromiseWorker.jsm:173:18) JS Stack trace: [email protected]:173:18 < [email protected]:292:9 < [email protected]:315:40 < [email protected]:933:23 < [email protected]:812:7 < this.PromiseWalker.scheduleWalkerLoop/<@Promise-backend.js:746:1 < [email protected]:770:1 < [email protected]:284:9
1465632860289addons.xpi-utilsWARNFailed to save XPI database: Error: Could not create new thread! (resource://gre/modules/PromiseWorker.jsm:173:18) JS Stack trace: [email protected]:173:18 < [email protected]:292:9 < [email protected]:315:40 < [email protected]:933:23 < [email protected]:812:7 < this.PromiseWalker.scheduleWalkerLoop/<@Promise-backend.js:746:1 < [email protected]:770:1 < [email protected]:284:9
1465632860290addons.xpi-utilsWARNFailed to save XPI database: Error: Could not create new thread! (resource://gre/modules/PromiseWorker.jsm:173:18) JS Stack trace: [email protected]:173:18 < [email protected]:292:9 < [email protected]:315:40 < [email protected]:933:23 < [email protected]:812:7 < this.PromiseWalker.scheduleWalkerLoop/<@Promise-backend.js:746:1 < [email protected]:770:1 < [email protected]:284:9
1465632860358DeferredSave.addons.jsonWARNWrite failed: Error: Could not create new thread! (resource://gre/modules/PromiseWorker.jsm:173:18) JS Stack trace: [email protected]:173:18 < [email protected]:292:9 < [email protected]:315:40 < [email protected]:933:23 < [email protected]:812:7 < this.PromiseWalker.scheduleWalkerLoop/<@Promise-backend.js:746:1 < [email protected]:770:1 < [email protected]:284:9
1465632860359addons.repositoryERRORSaveDBToDisk failed: Error: Could not create new thread! (resource://gre/modules/PromiseWorker.jsm:173:18) JS Stack trace: [email protected]:173:18 < [email protected]:292:9 < [email protected]:315:40 < [email protected]:933:23 < [email protected]:812:7 < this.PromiseWalker.scheduleWalkerLoop/<@Promise-backend.js:746:1 < [email protected]:770:1 < [email protected]:284:9
Segmentation fault
[pascal@firefox ~]$

[pascal@localhost ~]$ ulimit -aH
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 579483
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 65536
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) unlimited
cpu time               (seconds, -t) unlimited
max user processes              (-u) 579483
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
[pascal@localhost ~]$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 579483
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 32768
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 4096
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
[pascal@localhost ~]$ set /proc/*/task/*/cwd/.; echo $#
306
[pascal@localhost ~]$ prlimit
RESOURCE   DESCRIPTION                             SOFT      HARD UNITS
AS         address space limit                unlimited unlimited bytes
CORE       max core file size                         0 unlimited blocks
CPU        CPU time                           unlimited unlimited seconds
DATA       max data size                      unlimited unlimited bytes
FSIZE      max file size                      unlimited unlimited blocks
LOCKS      max number of file locks held      unlimited unlimited
MEMLOCK    max locked-in-memory address space     65536     65536 bytes
MSGQUEUE   max bytes in POSIX mqueues            819200    819200 bytes
NICE       max nice prio allowed to raise             0         0
NOFILE     max number of open files               32768     65536
NPROC      max number of processes                 4096    579483
RSS        max resident set size              unlimited unlimited pages
RTPRIO     max real-time priority                     0         0
RTTIME     timeout for real-time tasks        unlimited unlimited microsecs
SIGPENDING max number of pending signals         579483    579483
STACK      max stack size                       8388608 unlimited bytes

Edited on 6/13/16 at 10:24p to add:

Not a GUI problem. When I tried to su to the user today, that doesn't even work. Root is fine. I can ls, vi, create a new user, su to that user, everything works fine for that user, I exit and try to su to the problem user and no go. Bash kinda loaded the first time, but even exit didn't work. I had to reconnect to get back to root.

[root@firefox ~]# su - pascal
Last login: Sat Jun 11 03:08:47 CDT 2016 on pts/1
-bash: fork: retry: No child processes
-bash: fork: retry: No child processes
-bash: fork: retry: No child processes
-bash: fork: retry: No child processes
-bash: fork: Resource temporarily unavailable
-bash-4.2$ ls
-bash: fork: retry: No child processes
-bash: fork: retry: No child processes
-bash: fork: retry: No child processes
-bash: fork: retry: No child processes
-bash: fork: Resource temporarily unavailable
-bash: fork: retry: No child processes
-bash: fork: retry: No child processes
-bash: fork: retry: No child processes
-bash: fork: retry: No child processes
-bash: fork: Resource temporarily unavailable
-bash-4.2$ exit
logout
-bash: fork: retry: No child processes
-bash: fork: retry: No child processes
-bash: fork: retry: No child processes
-bash: fork: retry: No child processes
-bash: fork: Resource temporarily unavailable
-bash-4.2$ 

[root@firefox ~]# ls -l /
total 126
lrwxrwxrwx.    1 root root   7 Jan 28 23:53 bin -> usr/bin
---- snip ----
drwxr-xr-x.   19 root root  23 May 27 18:03 var
[root@firefox ~]# vi /etc/rc.local
[root@firefox ~]# useradd test
[root@firefox ~]# su - test
[test@firefox ~]$ cd
[test@firefox ~]$ ls -l
total 0
[test@firefox ~]$ ls -l /
total 126
lrwxrwxrwx.    1 root root   7 Jan 28 23:53 bin -> usr/bin
---- snip ----
drwxr-xr-x.   19 root root  23 May 27 18:03 var
[test@firefox ~]$ vi /etc/rc.local
[test@firefox ~]$ exit
logout
[root@firefox ~]# su - pascal
Last login: Mon Jun 13 22:12:12 CDT 2016 on pts/1
su: failed to execute /bin/bash: Resource temporarily unavailable
[root@firefox ~]#
Improve this question
2
  • With the new information, it looks like a problem either with X in general or with some specific libraries. Try running some non-demanding programs like xterm, xmag, ... Also try in an almost-empty environment: env -i HOME="$HOME" PATH=/usr/bin:/bin DISPLAY=:0 firefox (set XAUTHORITY as well if it's needed).
    – Gilles 'SO- stop being evil'
    Commented Jun 13, 2016 at 9:13
  • I don't understand what "problem either with X in general or with some specific libraries" would cause one UID to work and another to give those errors, when running the exact same commands on the same box in the same condition..
    – Pascal
    Commented Jun 14, 2016 at 0:42

4 Answers 4

Reset to default
6

nproc was the problem:

[root@localhost ~]# ps -eLf | grep pascal | wc -l
4068
[root@localhost ~]# cat /etc/security/limits.d/20-nproc.conf
# Default limit for number of user's processes to prevent
# accidental fork bombs.
# See rhbz #432903 for reasoning.

*          soft    nproc     4096
root       soft    nproc     unlimited
[root@localhost ~]#

man limits.conf states:

   Also, please note that all limit settings are set per login. They are
   not global, nor are they permanent; existing only for the duration of
   the session. One exception is the maxlogin option, this one is system
   wide. But there is a race, concurrent logins at the same time will not
   always be detected as such but only counted as one.

It appears to me that nproc is only enforced per login but counts globally. So a login with nproc 8192 and 5000 threads would have no problems, but a simultaneous login of the same UID with nproc 4096 and 50 threads would not be able to create more because the global count (5050) is above its nproc setting. 

[root@localhost ~]# ps -eLf | grep pascal | grep google/chrome | wc -l
3792
Improve this answer
3
  • Thanks for the update. What application was causing that number of processes?
    – fpmurphy
    Commented Jun 14, 2016 at 4:58
  • How did you find out?
    – Kashyap
    Commented Jun 14, 2016 at 13:23
  • 2
    I ended up using Google to figure out how to check each limit, 1 at a time. I eventually Googled "linux threads in use" which sent me to superuser.com/questions/80556/… which stated "ps -eLf will give you a list of all the threads and processes currently running on the system." I then used grep to filter for the one specific user and wc to count the lines.
    – Pascal
    Commented Jun 14, 2016 at 19:55
3

If you can't access the account at all, you'll have a hard time finding out what the problem is. But do check system or application logs, hopefully some program will have left a clue there (especially for a failed login attempt).

If you can run programs to experiment, you can tell which limit has been reached by attempting to increase each limited value and seeing when it works and when the attempt fails with EAGAIN. It's also possible to list the resources used for each value; I can't think of a utility that collects the data for all limits but there may well be one.

Assuming that the problem is a kernel limit, those are listed in the setrlimit man page. The ones that apply per user ID are:

  • RLIMIT_MEMLOCK — size of unswappable memory. Shouldn't prevent logging in, very few programs request unswappable memory.
  • RLIMIT_MSGQUEUE — size of message queues. Shouldn't prevent logging in, very few programs use message queues.
  • RLIMIT_NPROC — maximum number of processes. This one absolutely will prevent logins if it's reached. Increasing the limit in /etc/security/limits.conf won't affect the existing sessions, but it will affect new processes, so if the system administrator increases the value there, the user will be able to log in.
  • RLIMIT_SIGPENDING — maximum number of pending signals. Shouldn't prevent logging in, very few programs use sigqueue to enqueue signals.

So the limit on processes is the most likely one. If you have access to a running shell, you can confirm by trying to run a program; the error should be pretty distinctive:

$ ls
bash: fork: retry: No child processes
bash: fork: retry: No child processes
bash: fork: retry: No child processes
bash: fork: retry: No child processes
bash: fork: Resource temporarily unavailable

You can print out this limit with ulimit -u. If you have access to a shell running as the problematic user, and the user hasn't run any setuid program, you can list the processes that count against this limit with set /proc/*/task/*/cwd/.; echo $# (lists the kernel threads for which the user can read the cwd link, which means that the user has full control over the process).

Improve this answer
6
  • The affected user can always login, they just have problems running programs, both on existing logins and new ones.
    – Pascal
    Commented Jun 13, 2016 at 0:50
  • Exactly what message is being displaced to the user when a program fails to run?
    – fpmurphy
    Commented Jun 13, 2016 at 0:55
  • @Pascal If the user can log in, that's a completely different question! The traces you've posted don't point to a limit-related problem. Please post a new question with the new information.
    – Gilles 'SO- stop being evil'
    Commented Jun 13, 2016 at 9:08
  • @Gilles I'm wondering if you misread my second sentence. Perhaps "Not even a new login of that user can run new programs." would have been more clear.
    – Pascal
    Commented Jun 14, 2016 at 0:39
  • "set /proc/*/task/*/cwd/.; echo $#" appears to list processes only, not threads. In this case there were only 306 processes but 4068 threads. nproc limits threads.
    – Pascal
    Commented Jun 14, 2016 at 4:53
1

Try finding out what system calls are going on when OS tries to run your program and fails. Use strace

Hopefully it would be pretty straightforward. If not post the trace.

In general there is nothing else you can do but to try and gather more info.

Improve this answer
0

nproc you can ascertain with https://unix.stackexchange.com/a/289589/8337

open files -n you can ascertain with https://stackoverflow.com/a/21752125/32453

Not sure on the rest.

Maybe increase the limits one at a time until you hit gold? :)

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .