6

When I first borrowed an account on a UNIX system in 1990, the file limit was an astonishing 1024, so I never really saw that as a problem.

Today 30 years later the (soft) limit is a measly 1024.

I imagine the historical reason for 1024 was that it was a scarce resource - though I cannot really find evidence for that.

The limit on my laptop is (2^63-1):

$ cat /proc/sys/fs/file-max
9223372036854775807

which I today see as astonishing as 1024 in 1990. The hard limit (ulimit -Hn) on my system limits this further to 1048576.

But why have a limit at all? Why not just let RAM be the limiting resource?

I ran this on Ubuntu 20.04 (from year 2020) and HPUX B.11.11 (from year 2000):

ulimit -n `ulimit -Hn`

On Ubuntu this increases the limit from 1024 to 1048576. On HPUX it increases from 60 to 1024. In neither case is there any difference in the memory usage as per ps -edalf. If the scarce resource is not RAM, what is the scarce resource then?

I have never experienced the 1024 limit helping me or my users - on the contrary, it is the root cause for errors that my users cannot explain and thus cannot solve themselves: Given the often mysterious crashes they do not immediately think of ulimit -n 1046576 before running their job.

I can see it is useful to limit the total memory size of a process, so if it runs amok, it will not take down the whole system. But I do not see how that applies to the file limit.

What is the situation where the limit of 1024 (and not just a general memory limit) would help back in 1990? And is there a similar situation today?

5
  • You could increase that limit. With setrlimit(2) Commented Dec 22, 2020 at 8:04
  • can't make it as an answer, yet main benefit of "short" (i.e. 1024) limit is to prevent poorly writen shell/job/programming langage going astray.
    – Archemar
    Commented Dec 22, 2020 at 10:29
  • In neither case is there any difference in the memory usage as per ps -edalf Is that measuring kernel memory usage? The structures in question are in kernel memory, not in process memory. Commented Dec 22, 2020 at 11:20
  • @AndrewHenle What command would you run to show the amount of kernel memory in use? On Ubuntu? On HPUX?
    – Ole Tange
    Commented Dec 22, 2020 at 17:56
  • See related discussion on usenet from 1990. Commented Dec 28, 2020 at 15:14

3 Answers 3

3

@patbarron has still not posted his comments as an answer, and they are really excellent. So for anyone looking for the answer it is here.

He writes:

You can look at the source code from Seventh Edition, for example (minnie.tuhs.org/cgi-bin/utree.pl?file=V7/usr/sys/h/user.h) to see how this was implemented originally. "NOFILE" is the maximum number of open files per process, and it affects the sizes of data structures that are allocated per-process. These structures take up memory whether they're actually used or not. Again, mostly of historical interest, as it's not done this way anymore, but that might provide some additional background on where this came from.

The other constant, "NFILE", is the maximum number of open files in the entire system (across all processes/users), and the per-process table of open files contains pointers into the "files" structure: minnie.tuhs.org/cgi-bin/utree.pl?file=V7/usr/sys/conf/c.c. This is also a compile-time constant and sizes the system-wide open files table (which also consume memory whether they're actually used or not).

This explains that historically there was a reason. Each process would reserve NOFILE file descriptors - no matter whether they were used or not. When RAM is scarce you want to avoid reserving memory you do not use. Not only is RAM cheaper today, the reservation is no longer done this way.

It confirms my observations: I have been unable to find a single reason why you would keep ulimit -n at 1024 instead of raising it to the max: ulimit -n $(ulimit -Hn). It only takes up memory when the file descriptors are actually used.

0

As far as I know, yes, the file-max kernel hard limit was due to the memory allocation strategy (the memory for the inode structure was allocated beforehand). This strategy was common, intuitive and efficient (back then), and was shared between DOS, Windows, Linux and other OSes.

Nowadays, I believe that the huge number you see is the theoretical maximum (264-1), and the "real" file-max is allocated at runtime and can be set via ulimit (ulimit -Hn and ulimit -Sn). So, the "file-max" is just a sort of maximum value for ulimit, essentially meaningless - it means, "whatever, until ulimits runs out of RAM".

11
  • I like 2^64-1. But why having a limit anywhere lower today? Why not let RAM be the limiting resource? What is the reason for that? (My ulimit -Hn is 1M - and while that is not a problem today, it does not feel astonishing in the slightest).
    – Ole Tange
    Commented Dec 21, 2020 at 23:59
  • 1
    Well, I think that RAM is the limiting resource. You could easily increase the limit, but it makes little sense to have it maxed out by default since it would never fit everyone anyway (and you can't devote all the RAM to file structures). So, the default is a compromise, as always.
    – LSerni
    Commented Dec 22, 2020 at 0:07
  • 2
    At least back in the times mentioned in this answer ... when your system has a total of 128KW of memory, then increasing the open file limit by even just a bit can really add up...
    – patbarron
    Commented Dec 22, 2020 at 2:22
  • 1
    @OleTange - you can look at the source code from Seventh Edition, for example (minnie.tuhs.org/cgi-bin/utree.pl?file=V7/usr/sys/h/user.h) to see how this was implemented originally. "NOFILE" is the maximum number of open files per process, and it affects the sizes of data structures that are allocated per-process. These structures take up memory whether they're actually used or not. Again, mostly of historical interest, as it's not done this way anymore, but that might provide some additional background on where this came from.
    – patbarron
    Commented Dec 22, 2020 at 19:59
  • 1
    The other constant, "NFILE", is the maximum number of open files in the entire system (across all processes/users), and the per-process table of open files contains pointers into the "files" structure: minnie.tuhs.org/cgi-bin/utree.pl?file=V7/usr/sys/conf/c.c. This is also a compile-time constant and sizes the system-wide open files table (which also consume memory whether they're actually used or not).
    – patbarron
    Commented Dec 22, 2020 at 20:03
0

The common function used in networking code to monitor file descriptors, select(), only handles file descriptors up to 1023 in many implementations. While this function is generally considered obsolete in new code, software than uses it will not function properly with higher numbered file descriptors.

File descriptors are only known to the user process as integer values and functions which operate on sets of file descriptors were implemented by assuming a small fixed range of possible file descriptors and iterating through the entire range checking if each one was marked for processing. This ends up being extremely costly if the maximum file descriptor number is too large.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .