1

A custom application running on one of our development systems (Ubuntu 21.10 Desktop) is bombing out with the error socket: Too many open files. So I immediately checked the limits. Here's the output:

$ ulimit -n
65535

Now the reason this is strange is because the application is only using 1019 sockets when it bombs out. Given that there may be a few other file descriptors open I figured it is hitting a 1024 limit.

Why is the desktop imposing a 1024 limit when ulimit -n clearly says the limit is 65535?

Just to make this even more strange. I have two applications. An epoll based web scraper and a PACKET_MMAP based application that sends a SYN to multiple web servers to start a connection. The epoll application does not bomb out and yet uses much more than 1024 sockets. While it is the PACKET_MMAP based application using raw sockets that bombs out.

Also, the same applications tested on a server (also running Ubuntu) don't bomb out.

So whatever the problem is it is specific to the Desktop and to the raw sockets application.

EDIT:

Output of cat /proc/pid/limits as requested:

Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            8388608              unlimited            bytes     
Max core file size        0                    unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             31106                31106                processes 
Max open files            1024                 1048576              files     
Max locked memory         1025802240           1025802240           bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       31106                31106                signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us  

The output shows that soft limit of max open files is in fact 1024 contrary to the output of ulimit -n. What gives?

5
  • 2
    You need to describe how you run this application. Do you run it as a separate user? Do you run it from a process supervisor like systemd?
    – AlexD
    Jan 7, 2022 at 7:38
  • its easy to hit the limit, the problem in the past was, that the increase of the limit could only be done via recompile of the kernel. im unsure if this is still valid for the current versions
    – djdomi
    Jan 7, 2022 at 7:52
  • @AlexD I have revised the question and edited it. It's almost a complete rewrite of the question. I hope it answers your question. I think the answer is that the process is running under systemd.
    – James Read
    Jan 7, 2022 at 13:44
  • 1
    Show the output of cat /proc/1234/limits where 1234 is the PID of your program.
    – AlexD
    Jan 7, 2022 at 13:49
  • @AlexD I updated the question. The output does in fact show that there is a soft limit of 1024. Why is ulimit -n showing the wrong output? How can I fix this and raise the soft limit?
    – James Read
    Jan 7, 2022 at 15:04

3 Answers 3

5

Resource limits can be set differently for different users and different processes. Your ulimit shows specific limits for a specific user but systemd has its own default limits. You need to update your systemd unit file for your program to set file limits LimitNOFILE=1048576

See the documentation for details

4
  • Do you mean as detailed here superuser.com/questions/1200539/… ? I did that with no effect. Although I used DefaultLimitNOFILE=65535 rather than LimitNOFILE=1048576. Is there a difference? Or am I just editing the wrong files. I've been editing /etc/systemd/user.conf and /etc/systemd/system.conf
    – James Read
    Jan 7, 2022 at 15:34
  • 1
    You are changing the default settings. As I said, you need to change the limits for your specific program.
    – AlexD
    Jan 7, 2022 at 15:37
  • So which file do I need to add this to? I added this to /etc/systemd/user.conf and /etc/systemd/system.conf and rebooted and got the same problem. Output of cat /proc/pid/limits shows that Max open files limit is still 1024
    – James Read
    Jan 8, 2022 at 18:02
  • You said that you are running your program from systemd. You should find a file under /etc/systemd/ which starts your specific program and add the LimitNOFILE line.
    – AlexD
    Jan 9, 2022 at 5:42
0

This is how I solved this problem with some C code. I added the following lines of code to my application:

#include <sys/resource.h>

...

    struct rlimit lim;
    
    getrlimit(RLIMIT_NOFILE, &lim);
    printf("Soft: %d Hard: %d\n", (int)lim.rlim_cur, (int)lim.rlim_max);
    lim.rlim_cur = lim.rlim_max;
    
    
    if (setrlimit(RLIMIT_NOFILE, &lim) == -1) {
        printf("rlimit failed\n");
        return -1;
    }
    getrlimit(RLIMIT_NOFILE, &lim);
    printf("New Soft: %d New Hard: %d\n", (int)lim.rlim_cur, (int)lim.rlim_max);

But I still have no idea how to do this by editing configuration files on the system. However, this answer can still be useful for those who are satisfied with solving the problem with C code.

0

When You want to change the limits for a service managed by systemd in Ubuntu Linux, You'll need to configure this in an override.conf file. I'm using mariadb as an example, but this can be generalized for all services.

mkdir /etc/systemd/system/mariadb.service.d
vi /etc/systemd/system/mariadb.service.d/override.conf

In this file You'll enter the following lines:

[Service]
LimitNOFILE=65535

After this, You'll need to reload the Unit declaration

systemctl daemon-reload

and restart the service

systemctl restart mariadb.service

Now You'll be able to administratively control the process limits. Systemd can manage the following limits with this method:

Directive ulimit equivalent Unit
LimitCPU= ulimit -t Seconds
LimitFSIZE= ulimit -f Bytes
LimitDATA= ulimit -d Bytes
LimitSTACK= ulimit -s Bytes
LimitCORE= ulimit -c Bytes
LimitRSS= ulimit -m Bytes
LimitNOFILE= ulimit -n Number of File Descriptors
LimitAS= ulimit -v Bytes
LimitNPROC= ulimit -u Number of Processes
LimitMEMLOCK= ulimit -l Bytes
LimitLOCKS= ulimit -x Number of Locks
LimitSIGPENDING= ulimit -i Number of Queued Signals
LimitMSGQUEUE= ulimit -q Bytes
LimitNICE= ulimit -e Nice Level
LimitRTPRIO= ulimit -r Realtime Priority
LimitRTTIME= No equivalent Microseconds

From the systemd documentation https://www.freedesktop.org/software/systemd/man/systemd.exec.html

You must log in to answer this question.