10

Is there a (technical or practical) limit to how large you can configure the maximum number of open files in Linux? Are there some adverse effects if you configure it to a very large number (say 1-100M)?

I'm thinking server usage here, not embedded systems. Programs using huge amounts of open files can of course eat memory and be slow, but I'm interested in adverse effects if the limit is configured much larger than necessary (e.g. memory consumed by just the configuration).

10

I suspect the main reason for the limit is to avoid excess memory consumption (each open file descriptor uses kernel memory). It also serves as a safeguard against buggy applications leaking file descriptors and consuming system resources.

But given how absurdly much RAM modern systems have compared to systems 10 years ago, I think the defaults today are quite low.

In 2011 the default hard limit for file descriptors on Linux was increased from 1024 to 4096.

Some software (e.g. MongoDB) uses many more file descriptors than the default limit. The MongoDB folks recommend raising this limit to 64,000. I've used an rlimit_nofile of 300,000 for certain applications.

As long as you keep the soft limit at the default (1024), it's probably fairly safe to increase the hard limit. Programs have to call setrlimit() in order to raise their limit above the soft limit, and are still capped by the hard limit.

See also some related questions:

  • 6
    This hasn't actually answered the question, though, which asked if there was a technical or practical limit to how high one can set the hard limit. There is, but this answer does not mention it at all. – JdeBP Jan 2 '17 at 8:19
  • I'm finding it impossible to raise the limit beyond about 1million. I think it might be hard-coded in the kernel, because despite changing many configurations, I can't raise beyond this. superuser.com/questions/1468436/… – Pavel Komarov Aug 7 '19 at 16:16
3

The impact wouldn't normally be observable, but the kernel's IO module will have to take care of all those open file descriptors and they could also have an impact on cache efficiency.

Such limits have the advantage of protecting the user from their own (or third parties') mistakes. For example, if you run a small program or script that forks indefinitely, it will eventually block on one of the ulimits and therefore prevent a more intense (possibly unrecoverable) computer freeze.

Unless you have precise reasons to increase any of those limits, you should avoid it and sleep better.

2

It is technically limited to maximum value of unsigned long (C Lang) i.e. 4,294,967,295

Reference : fs.h file

/* And dynamically-tunable limits and defaults: */
struct files_stat_struct {
  unsigned long nr_files;   /* read only */
  unsigned long nr_free_files;  /* read only */
  unsigned long max_files;    /* tunable THIS IS OUR VALUE */
};
  • 2
    Do you have any references for this? – Tim May 25 '19 at 10:15
  • Also that's the max value for 32-bit signed integer, 32-bit unsigned integer max value is 4,294,967,295. – Sampo May 25 '19 at 14:59
  • You are right, Sampo. My mistake. – Leonard T May 26 '19 at 11:04
0

I think your concern is understandable but most likely Linux will not consume much memory for configured (but not used file descriptors) :)

I can't remember such a problem in my professional career for the last 10 years.

Regards.

0

Quiet late but this should help everyone else to get the answer for this question. The practical limit for number of open files in linux can also be counted using maximum number of file descriptor a process can open.

I have seen limits being changed from system to system. From the getlimit man page you can see that RLIMIT_NOFILE-1 specifies the limits internally.

To check RLIMIT_NOFILE value you can use the below statement to get a tuple

python -c "import resource; print(resource.getrlimit(resource.RLIMIT_NOFILE))"

The tuple returns results as (Soflimit, hardlimit). For me running on multiple systems results are like below

(1024, 1048576) # on UBUNTU linux 
(65536, 65536)  # on amazon linux 
(1024, 9223372036854775807) # on macos 

Note: 9223372036854775807 this number simply means infinity. You will always reach other resource limits before you ever hit this. If you have do modify the hardlimit on a system beyond what it is, you will have to modify kernel params.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.