27

I need some assistance grasping what I'm sure is a fundamental concept in Linux: the limit for open files. Specifically, I'm confused on why open sockets can count towards the total number of "open files" on a system.

Can someone please elaborate on the reason why? I understand that this probably goes back to the whole "everything is a file" principle in Linux but any additional detail would be appreciated.

32

The limit on "open files" is not really just for files. It's a limit on the number of kernel handles a single process can use at one time. Historically, the only thing that programs would typically open a lot of were files, so this became known as a limit on the number of open files. There is a limit to help prevent processes from say, opening a lot of files and accidentally forgetting to close them, which will cause system-wide problems eventually.

A socket connection is also a kernel handle. So the same limits apply for the same reasons - it's possible for a process to open network connections and forget to close them.

As noted in the comments, kernel handles are traditionally called file descriptors in Unix-like systems.

  • 23
    "Kernel handles" is a Windows terminology. You'd rather refer to "file descriptors" which is how these entities are generally called with Unix & Linux. – jlliagre Sep 24 '14 at 22:35
  • 11
    This answer hedges too much. Sockets are files. They provide access to streams of bytes through the read/write interface, which is the heart of what it means to be a file. – Wumpus Q. Wumbley Sep 25 '14 at 0:49
  • 4
    @WumpusQ.Wumbley, but then you have the shutdown(2) syscall on them, but not on files, and you can't read from a socket using cat -- that's the reason netcat has been created. I'd say that (luckily) sockets in Unix-like kernels behave like files in terms of I/O, but the similarity ends right there. (Honestly, I'd also like to hear from someone with Plan 9 experience as I've heard they got unification of these things farther than traditional unices). – kostix Sep 25 '14 at 8:23
  • @MikeB, this book should get you up to speed with most Unix-related concepts. Highly recommended. – kostix Sep 25 '14 at 8:26
  • 3
    The "everything is a file" idea means that "file" is an abstract data type with many subtypes. Most of the subtypes support extra methods in addition to the basic stuff that all files support. sockets have lots of extras. block devices and regular files have seek. directories are really weird (write doesn't work and if read works, it's not useful). The presence of extra methods doesn't mean these things aren't part of the general category of things we call "files". – Wumpus Q. Wumbley Sep 25 '14 at 19:40
23

The reason why TCP/IP sockets use file descriptors is that, when the sockets interface was first designed and implemented (in BSD Unix, in 1983), its designers felt that a network connection was analogous to a file - you can read, write, and close both, and that it would fit well with the Unix idea of "everything is a file".

Other TCP/IP network stack implementations didn't necessarily integrate with their OS's file-I/O subsystem, an example being MacTCP. But because the BSD sockets interface was so popular, even these other implementations chose to replicate the socket API with its Unix-like functions, so you got "file descriptors", only used for TCP/IP communication, on systems that didn't otherwise have file descriptors.

The other part of your question is why is there a limit? It's because the quickest way to implement a file descriptor lookup table is with an array. Historically, the limit was hard-coded into the kernel.

Here's the code in Unix release 7 (1979) with a hard-coded limit 20 file descriptors per process:

  • user.h: struct file *u_ofile[NOFILE]
  • param.h: #define NOFILE 20

By comparison, Linux dynamically allocates space for a process's file descriptor table. The absolute limit defaults to 8192, but you can set this to whatever you like. My system lists 191072 in /proc/sys/fs/file-max.

Despite there being no absolute limit in Linux any more, nonetheless we don't want to let programs go crazy, so the administrator (or the distribution packager) generally sets resource limits. Take a look at /etc/security/limits.conf, or run ulimit -n.

  • One of the best answers in this topic, thanks – user859375 Jul 13 '18 at 6:31
5

Files are not just files on disk or in memory; they are streams of data, of which those are but two examples.

Remote endpoints are a third example, and you interact with those using sockets.

  • 2
    Welcome to U&L.SE. I like this answer. – eyoung100 Sep 25 '14 at 17:21

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.