Change the resource limits (ulimit / rlimit) of a running process

Question

I have a long-running process that is hitting a resource limit, such as the maximum number of open files.

I don't want to kill it.

Usually, you'd do:

(stop service)
ulimit -n <new limit>
(start service)

Is there a way to avoid having to stop and start the service and increase the limits?

Answer 1

I've figured it out.

On some kernels (e.g. 2.6.32+), at least on CentOS/RHEL, you can change the resource limits of a running process using /proc/<pid>/limits, e.g.:

$ grep "open files" /proc/23052/limits
Limit                     Soft Limit           Hard Limit           Units     
Max open files            1024                 4096                 files     

To change the maximum open files to a soft limit of 4096, hard limit of 8192:

echo -n "Max open files=4096:8192" > /proc/23052/limits

This gives:

$ grep "open files" /proc/23052/limits
Limit                     Soft Limit           Hard Limit           Units     
Max open files            4096                 8192                 files

Note the -n in echo -n - without that, you'll get an "invalid argument" error.

The above doesn't always work, so

Another option is prlimit command, introduced with util-linux 2.21 allows you to read and change the limits of running processes.

This is a followup to the writable /proc/<pid>/limits, which was not integrated in mainline kernel. This solution should work.

$ prlimit  --nofile --output RESOURCE,SOFT,HARD --pid 23052
RESOURCE SOFT HARD
NOFILE   1024 4096

Set the limits:

$ prlimit --nofile=4096:8192 --pid 23052

Confirm:

$ prlimit  --nofile --output RESOURCE,SOFT,HARD --pid 23052RESOURCE SOFT HARD
NOFILE   4096 8192

$ grep "open files" /proc/23052/limits
Limit                     Soft Limit           Hard Limit           Units     
Max open files            4096                 8192                 files     

Answer 2

You can do it with prlimit before running:

inFilesCount=1100 # greater than default 1024
# with N in file, you need to open N+1 file where output file is the 1
totalFilesCount="$((inFilesCount+1))"
# when soft limit is 1024, sort can only open up to 1021 files, maybe stdout, stdin, stderr are already open
prlimit --nofile="$((totalFilesCount+3))" sort --batch-size="$totalFilesCount" -m -u -o output.file *.csv

Or you do it while running

prlimit --nofile="$((totalFilesCount+3))" --pid=xxx