42

How can I determine or set the size limit of /etc/hosts? How many lines can it have?

  • 62
    ... I think at the point where this becomes an actual worry rather than an academic curiosity you have far passed the point where it would be wiser to set up a private DNS server in terms of performance and maintainability. – Shadur Aug 28 '17 at 17:17
  • 5
    I'm surprised you are asking. Why do you expect some precise and hard meaningful size limit on configuration files? – Basile Starynkevitch Aug 29 '17 at 4:22
  • 3
    @BasileStarynkevitch I asked because a ~2 MB hosts file wasn't working on my router, but the issue was that I didn't sighup dnsmasq to re-read the hosts file. – Geremia Aug 29 '17 at 17:18
  • 2
    @Geremia my hosts file is 500M, 15,000+ entries, no problems. It is always best to determine the root cause of your problem and solve it first – bsd Aug 29 '17 at 22:35
  • 3
    Are you using your host file to block IP addresses? If so you should be using iptables with ipset. Using iptables alone will cause a major performance hit, however with ipset a list of nearly 500,000 ips has negligible impact. – cybernard Aug 30 '17 at 0:47
57

Problematical effects include slow hostname resolution (unless the OS somehow converts the linear list into a faster-to-search structure?) and the potential for surprising interaction with shell tab completion well before any meaningful file size is reached.

For example! If one places 500,000 host entries in /etc/hosts

# perl -E 'for (1..500000) { say "127.0.0.10 $_.science" }' >> /etc/hosts

for science, the default hostname tab completion in ZSH takes about ~25 seconds on my system to return a completion prompt (granted, this is on a laptop from 2008 with a 5400 RPM disk, but still).

22

I don't think it has a size limit in terms of number of lines.

Before DNS (which was brought into use in 1985), it was this file that served as the only means for doing host name lookups, so I presume that this means that the file should be able to have thousands or at least hundreds of entries to be able to support the most well-connected pre-1985 Internet nodes.

Here's an example from 1985 (the format has changed somewhat): http://jim.rees.org/apollo-archive/hosts.txt This file has 1680 lines out of which 1325 are host lines. The remaining 355 lines are blank, comments, networks or gateways1.

The only real limit that I could find was that on some systems, the individual lines are limited to less than BUFSIZ characters (1024 on my OpenBSD machine).

If you have more than a couple of handfuls of entries in /etc/hosts, you should consider setting up a local name server instead, but that's my personal opinion.


1 Thanks to Jeff Schaller for digging this up.

  • Before DNS, I don't think it was common to convert the full Internet host table to /etc/hosts format. Most Unix systems weren't even on the Internet, and even if a machine was, it didn't need a complete host table, just the handful of the machines it needed to talk to. I'd be surprised if there were many machines with more than 100 entries. – Barmar Aug 30 '17 at 23:15
13

How can I determine the size limit of /etc/hosts?

It is a regular file, so the limit would correspond to the underlying filesystem's limits (which would itself be capped by the number of disks behind it), minus the space used by any other files in the same (probably root (/)) filesystem:

  • ext2/3: 2 TiB
  • ext4: 16 TiB (with the default 4KiB block size)
  • xfs: 500 TiB

How can I set the size limit of /etc/hosts?

Since it's a manually-edited file, only manually:

sed -i '100,$d' /etc/hosts

(to remove lines 100 and beyond).

  • 3
    True in principle, but the file system limits (e.g. terabytes) are practically largely irrelevant. – Basile Starynkevitch Aug 29 '17 at 4:22
11

Size limits only apply when allocating static buffers. gethostbyname(3), which parses the entries in /etc/hosts, does not allocate static buffers - and never has. The original 1983 release of BSD 4.3 algorithm shows an open-file, while parse line, close-file pattern:

sethostent(0);
while (p = gethostent()) {
    if (strcmp(p->h_name, name) == 0)
        break;
    for (cp = p->h_aliases; *cp != 0; cp++)
        if (strcmp(*cp, name) == 0)
            goto found;
}
found:
endhostent();

Modern implementations retain this heritage in all essentials.

Anyway, internally, the *hostent family of functions store a file pointer to the current line in the file. sethostent opens the file and sets the file pointer position. gethostent gets data and advances the pointer. endhostent closes the file pointer. The GNU C Library offers a thorough reference on these functions.

As you might guess from the implementation, entries occuring earlier in the file resolve faster. If your hosts file is huge, this comes into play.

So, no matter how big the file is, the OS will consume it. Eventually, though, you'll hit filesystem limits (per Jeff Schaller's answer). You also have maximum line size limits (per Kusalananda's answer). But, in the end, you can make it as big as you want. But please, don't.

  • 2
    Using the glibc and linux, this is not so straightforward unfortunately. If you call gethostbyname and the system is setup accordingly (default on many systems) then it will instead of reading /etc/host call the nscd. I have no idea if the nscd will cache only hits in the file or try to cache it as a whole. In the later case, you would have a ram limitation for the file size (assuming that the nscd config allows that much entries) – PlasmaHH Aug 29 '17 at 12:08
  • 1
    The 4.3BSD version is here. It supported a dbm-hashed version of /etc/hosts. IIRC, dbm imposed some size limits that could cause attempts to create a hashed db to fail. – Mark Plotnick Aug 31 '17 at 11:53
2

... I've racked my brain and for the life of me I can't think of a single situation or circumstance where you'd approach any kind of size limit issue in /etc/hosts -- you'd run into practical problems like the severe performance hit to the getaddrinfo() family of system calls that all have to consult the file before deciding whether to send out a DNS query, to say nothing of the problems maintaining a flat text file that size.

I suspect that what we have here is a failure to communicate at a higher level. What problem are you trying to solve with a gigantic /etc/hosts file? I'm almost certain that there's a better solution than this.

  • 8
    Some people use a hosts file to blacklist ads/malware/tracking/etc. There are curated lists on the internet, the one I use is 41k lines and 1.1MB in size. – Bert Aug 29 '17 at 14:57
  • It would probably be better for performance to use dnsmasq for that - see e.g. dnsgate (which I haven't tried). – reinierpost Aug 30 '17 at 10:20
  • Mine has 1.7MB with 57k lines. You can get your huge /etc/hosts file from hostsfile.org – styrofoam fly Aug 30 '17 at 11:30
  • @reinierpost that's pretty insightful actually! – Jeff Schaller Sep 5 '17 at 11:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.