What is the /etc/hosts size limit?

Question

How can I determine or set the size limit of /etc/hosts? How many lines can it have?

Answer 1

Problematical effects include slow hostname resolution (unless the OS somehow converts the linear list into a faster-to-search structure?) and the potential for surprising interaction with shell tab completion well before any meaningful file size is reached.

For example! If one places 500,000 host entries in /etc/hosts

# perl -E 'for (1..500000) { say "127.0.0.10 $_.science" }' >> /etc/hosts

for science, the default hostname tab completion in ZSH takes about ~25 seconds on my system to return a completion prompt (granted, this is on a laptop from 2009 with a 5400 RPM disk, but still).

Answer 2

I don't think it has a size limit in terms of number of lines.

Before DNS (which was brought into use in 1985), it was this file that served as the only means for doing host name lookups, so I presume that this means that the file should be able to have thousands or at least hundreds of entries to be able to support the most well-connected pre-1985 Internet nodes.

Here's an example from 1985 (the format has changed somewhat): http://jim.rees.org/apollo-archive/hosts.txt This file has 1680 lines out of which 1325 are host lines. The remaining 355 lines are blank, comments, networks or gateways¹.

The only real limit that I could find was that on some systems, the individual lines are limited to less than BUFSIZ characters (1024 on my OpenBSD machine).

If you have more than a couple of handfuls of entries in /etc/hosts, you should consider setting up a local name server instead, but that's my personal opinion.

---

¹ _{Thanks to Jeff Schaller for digging this up.}

Answer 3

How can I determine the size limit of /etc/hosts?

It is a regular file, so the limit would correspond to the underlying filesystem's limits (which would itself be capped by the number of disks behind it), minus the space used by any other files in the same (probably root (/)) filesystem:

• ext2/3: 2 TiB
• ext4: 16 TiB (with the default 4KiB block size)
• xfs: 500 TiB

How can I set the size limit of /etc/hosts?

Since it's a manually-edited file, only manually:

sed -i '100,$d' /etc/hosts

(to remove lines 100 and beyond).

Answer 4

Size limits only apply when allocating static buffers. gethostbyname(3), which parses the entries in /etc/hosts, does not allocate static buffers - and never has. The original 1983 release of BSD 4.3 algorithm shows an open-file, while parse line, close-file pattern:

sethostent(0);
while (p = gethostent()) {
    if (strcmp(p->h_name, name) == 0)
        break;
    for (cp = p->h_aliases; *cp != 0; cp++)
        if (strcmp(*cp, name) == 0)
            goto found;
}
found:
endhostent();

Modern implementations retain this heritage in all essentials.

Anyway, internally, the *hostent family of functions store a file pointer to the current line in the file. sethostent opens the file and sets the file pointer position. gethostent gets data and advances the pointer. endhostent closes the file pointer. The GNU C Library offers a thorough reference on these functions.

As you might guess from the implementation, entries occuring earlier in the file resolve faster. If your hosts file is huge, this comes into play.

So, no matter how big the file is, the OS will consume it. Eventually, though, you'll hit filesystem limits (per Jeff Schaller's answer). You also have maximum line size limits (per Kusalananda's answer). But, in the end, you can make it as big as you want. But please, don't.

Answer 5

... I've racked my brain and for the life of me I can't think of a single situation or circumstance where you'd approach any kind of size limit issue in /etc/hosts -- you'd run into practical problems like the severe performance hit to the getaddrinfo() family of system calls that all have to consult the file before deciding whether to send out a DNS query, to say nothing of the problems maintaining a flat text file that size.

I suspect that what we have here is a failure to communicate at a higher level. What problem are you trying to solve with a gigantic /etc/hosts file? I'm almost certain that there's a better solution than this.