When spinning up multiple docker containers in which I run
npm ci, I start getting pthread_create: Resource temporarily unavailable errors (less than 5 docker containers can run fine). I deduce there is some kind of thread limit somewhere, but I cannot find which one is blocking here.
- a Jenkins instance spins up docker containers for each build (connection through ssh into this docker container).
- in each container some build commands are run; I see the error often when using
npm cisince this seems to create quite some threads; but I don't think the problem is related to
- all docker containers run on a single docker-host. It's specifications:
- Intel(R) Xeon(R) Gold 5118 CPU @ 2.30GHz with 12 cores, 220 GB RAM
- Centos 7
- Docker version 18.06.1-ce, build e68fc7a
- systemd version 219
- kernel 3.10.0-957.5.1.el7.x86_64
I can see the error under different forms:
- jenkins failing to contact the docker container; errors like: java.lang.OutOfMemoryError: unable to create new native thread
git clonefailing inside the container with ERROR: Error cloning remote repo 'origin' ... Caused by: java.lang.OutOfMemoryError: unable to create new native thread
npm cifailing inside the container with node: pthread_create: Resource temporarily unavailable
Things I have investigated or tried
I looked quite a lot a this question.
- docker-host has
systemdversion 219 and is hence does not have the
- number of threads (
ps -elfT | wc -l) is typically 700, but with multiple containers running I have seen it climb to 4500.
- all builds run as some user with pid 1001 inside the docker container; however there is no user with pid 1001 on the docker-host so I don't know which limits apply to this user.
- I have already increased multiple limits for all users in
- I created a dummy user with uid 1001 on docker-host and made sure it had also
nproclimit set to unlimited. Logging onto that user
ulimit -u= unlimited. This still didn't solve the problem
* soft nproc unlimited * soft stack 65536 * soft nofile 2097152
ulimit -a as root:
core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 899154 max locked memory (kbytes, -l) 1048576 max memory size (kbytes, -m) unlimited open files (-n) 1048576 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 65536 cpu time (seconds, -t) unlimited max user processes (-u) 899154 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited
limits of my dockerd process (cat
/proc/16087/limits where 16087 is pid of dockerd)
Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size unlimited unlimited bytes Max core file size unlimited unlimited bytes Max resident set unlimited unlimited bytes Max processes unlimited unlimited processes Max open files 65536 65536 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 899154 899154 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us