4

I was successfully using a small (20,000 entries) zone file with bind9 server, but today my data provider sent an update which caused the zone file to become 300,000+ entries large (30Mb+).

The problem is the server would not start with this zone file. The named-checkconf would not report any errors. No log messages are available (or I could not log them properly).

I would like to know if bind9 is capable of handling large configuration files and if yes how do I fix it. If no I would like to know if there are any workarounds for this issue. Maybe it's possible to store entries in a database?

The zone file I'm trying to use can be downloaded from here.

Update:

service bind9 status showed some information which may be relevant:

adjusted limit on open files from 4096 to 1048576
found 1 CPU, using 1 worker thread
using 1 UDP listener per interface
using up to 4096 sockets
loading configuration from '/etc/bind/named.conf'

I'm not quite sure how to interpret or use this information... Any ideas? Also I was not able to find where bind9 logs are located: /var/log/ has no bind9 entries. Can anybody tell me where they are located on Debian Jessie?

  • I have downloaded the file, and frankly it is unbelievable. Creating 350k+ domains in BIND? Not that it helps on my answer, how much RAM have you got free in that DNS server? This is an advert blacklist, right? – Rui F Ribeiro Jan 7 '16 at 13:21
  • I've downloaded the file into my /etc/bind directory and named starts just fine. You must have some configuration error, check the relevant log file (e.g. syslog, messages, daemon.log in /var/log/ depending on your OS). Try named-checkconf to check your config. – wurtel Jan 7 '16 at 13:23
  • 3
    For checking a zone, you want named-checkzone and not named-checkconf. The latter only validates that your configuration files are valid; the former validates that a zone is valid (when used with a particular zone name). Also, you can pass options to the BIND daemon to increase its verbosity and run in the foreground; try doing that. – a CVn Jan 7 '16 at 13:31
  • I added to my answer, sudo grep named /var/log/syslogfor seeing BIND logs – Rui F Ribeiro Jan 7 '16 at 21:11
7

I have seen your zone file: it appears to be a list of more than 350k domains, at the moment, where it is defined the local BIND server as the master. The domains are with the following format:

zone "xxxx.com" { type master; notify no; file "null.zone.file"; };

As per memory requirements, I would say as a ballpark figure you might need around 40MB-80MB of free RAM for that as domain tables are loaded in memory. (albeit I would feel more comfortable with 200MB at least)

Unless the server is severely constrained in RAM, it seems a bit improbable, but it could happen.

I also have noticed there are underscores ("_") in the name of several domains. Having underscores in DNS RR breaks a couple of RFCs (RFC 952 and RFC 1123), and you need to add to the BIND options section the directive:

check-names master ignore;

As for the format and method being used for blacklisting domains. From version 9.8 onwards, BIND supports what is known as Response Policy Zones (RPZ), that were created specifically for blacklisting domains.

Several (commercial) blacklist providers follow nowadays that format. (I myself use RPZ both at work and at home).

Using RPZ should make more sense and also means a lighter load, and as such, if you are paying the service, I would advise you to contact your supplier to know how to use it. The RPZ format also supports to some extent wildcards, which would mean a much smaller blacklist file.

An alternative is also to process the file with a script to alter it to RPZ format.

I will leave here relevant links about RPZ and official RPZ providers:

https://dnsrpz.info

and a tutorial how to configure RPZ:

http://www.zytrax.com/books/dns/ch9/rpz.html

As you may have noted, with the current configuration, you will also have a lot of open files; hence I recommend again using RPZ.

For dealing with more open files, in large email, DNS or HTTP servers, the limits have often to be raised.

The situation is not so bad as it used to be with older kernels, but nonetheless I do recommend raising the limits.

Edit /etc/sysctl.conf and modify/add the directive fs.file-max for the global limit of open files:

fs.file-max=500000

For applying the new file limit without rebooting, you need to run:

sudo sysctl -p

And for the files limits per process, edit, /etc/security/limits.conf:

* - nofile 400000

To apply the file limits per process, either logout and login, or run:

sudo ulimit -n 400000

After raising these two limits, you need to restart BIND:

sudo service bind9 restart

To convert your file to RPZ format, you run:

cat bind | tr -d \"  | awk ' { print $2" CNAME ." } ' > /etc/bind/rpz.db

The script will convert the entries to the following format:

zeus.developershed.com CNAME .
zeusclicks.com CNAME .
zintext.com CNAME .

Add in the options section of named:

response-policy { zone "rpz"; };

Create the declaration of the RPZ zone:

zone "rpz" {
  type master;
  file "/etc/bind/rpz.db";
};

Add to the top of /etc/bind/rpz.db file:

$TTL    604800
@       IN      SOA     localhost. root.localhost. (
                          2         ; Serial
                     604800         ; Refresh
                      86400         ; Retry
                    2419200         ; Expire
                     604800 )       ; Negative Cache TTL

@       IN      NS      your_dns_fqdn.

Deconfigure that DNS file of yours and restart your BIND server. Evidently the RPZ file can be optimised with wildcards and made much shorter, however even without that optimisation now you won't need so much open files.

As for consulting BIND/DNS logs, they are together with the system logs in /var/log/syslog with the tag named. You can use the command:

sudo grep named /var/log/syslog
  • Thank you for your assistance. I definitely have couple of hundreds megabytes of RAM on my machine plus a large swap. So this is probably has nothing to do with RAM. I checked service bind9 status and it reported some information which I added to the question: adjusted limit on open files from 4096 to 1048576 ... using up to 4096 sockets. Does it mean that bind reaches some system limit? Can I alter this limit somehow? – Kolyunya Jan 7 '16 at 15:53
  • Added to the answer. – Rui F Ribeiro Jan 7 '16 at 16:09
  • I would like to stress out the situation is not sustainable. Consider raising the file limits a stop gap temporary measure, configuring RPZ, and writing a program/script to convert the blacklist to RPZ format. Bear in mind you have got to have BIND 9.10+. – Rui F Ribeiro Jan 7 '16 at 16:26
  • I documented how to convert to RPZ. – Rui F Ribeiro Jan 7 '16 at 19:19
  • Thanks again, but we don't have version 9.10 available yet... – Kolyunya Jan 7 '16 at 21:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.