ulimit - one service affects another

Question

We are hosting 2 services on our RHEL 7.4 server: Jupyterlab and RStudio. Jupyterlab sessions of individual users tend to open a lot of files:

# lsof|grep e613664|wc -l
31581

Our problem is that the users who have an active Jupyterlab session (and therefore many open files) can't use RStudio. they're getting various 'cannot fork', 'resource unavailable' errors, or can't even use RStudio. As soon as we kill their Jupyterlab session, RStudio starts working fine for them.

Initially, I thought that the problem is caused by the large number of open files. The number of processes for each user is low, couple of dozens at most. Therefore, I've changed the ulimit -n value globally to 65k. our current ulimit settings:

$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 31136
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 65536
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 16384
cpu time               (seconds, -t) unlimited
max user processes              (-u) 4096
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

But the problems didn't go away. Are there any other per-user limits that I should be looking at?

Answer 1

It turns out that this is an RStudio-related issue. under /etc/rstudio/profiles we needed to increase the parameter max-processes (equivalent to nproc). We were limiting per-user processes to 120 to limit heavy parallelisation. Although the users suffering from this problem only had a couple of running processes at the OS level, it turned out to be a low number and after increasing the value to max-processes = 800, the problem disappeared. More documented here: https://docs.rstudio.com/ide/server-pro/r-sessions.html under clause 5.2.5.2.