Large zone file for bind9 : ad-blocking

Question

I was successfully using a small (20,000 entries) zone file with bind9 server, but today my data provider sent an update which caused the zone file to become 300,000+ entries large (30Mb+).

The problem is the server would not start with this zone file. The named-checkconf would not report any errors. No log messages are available (or I could not log them properly).

I would like to know if bind9 is capable of handling large configuration files and if yes how do I fix it. If no I would like to know if there are any workarounds for this issue. Maybe it's possible to store entries in a database?

The zone file I'm trying to use can be downloaded from here.

Update:

service bind9 status showed some information which may be relevant:

adjusted limit on open files from 4096 to 1048576
found 1 CPU, using 1 worker thread
using 1 UDP listener per interface
using up to 4096 sockets
loading configuration from '/etc/bind/named.conf'

I'm not quite sure how to interpret or use this information... Any ideas? Also I was not able to find where bind9 logs are located: /var/log/ has no bind9 entries. Can anybody tell me where they are located on Debian Jessie?

Answer 1

I have seen your zone file: it appears to be a list of more than 350k domains, at the moment, where it is defined the local BIND server as the master. The domains are with the following format:

zone "xxxx.com" { type master; notify no; file "null.zone.file"; };

As per memory requirements, I would say as a ballpark figure you might need around 40MB-80MB of free RAM for that as domain tables are loaded in memory. (albeit I would feel more comfortable with 200MB at least)

Unless the server is severely constrained in RAM, it seems a bit improbable, but it could happen.

I also have noticed there are underscores ("_") in the name of several domains. Having underscores in DNS RR breaks a couple of RFCs (RFC 952 and RFC 1123), and you need to add to the BIND options section the directive:

check-names master ignore;

As for the format and method being used for blacklisting domains. From version 9.8 onwards, BIND supports what is known as Response Policy Zones (RPZ), that were created specifically for blacklisting domains.

Several (commercial) blacklist providers follow nowadays that format. (I myself use RPZ both at work and at home).

Using RPZ should make more sense and also means a lighter load, and as such, if you are paying the service, I would advise you to contact your supplier to know how to use it. The RPZ format also supports to some extent wildcards, which would mean a much smaller blacklist file.

An alternative is also to process the file with a script to alter it to RPZ format.

I will leave here relevant links about RPZ and official RPZ providers:

https://dnsrpz.info

and a tutorial how to configure RPZ:

http://www.zytrax.com/books/dns/ch9/rpz.html

As you may have noted, with the current configuration, you will also have a lot of open files; hence I recommend again using RPZ.

For dealing with more open files, in large email, DNS or HTTP servers, the limits have often to be raised.

The situation is not so bad as it used to be with older kernels, but nonetheless I do recommend raising the limits.

Edit /etc/sysctl.conf and modify/add the directive fs.file-max for the global limit of open files:

fs.file-max=500000

For applying the new file limit without rebooting, you need to run:

sudo sysctl -p

And for the files limits per process, edit, /etc/security/limits.conf:

* - nofile 400000

To apply the file limits per process, either logout and login, or run:

sudo ulimit -n 400000

After raising these two limits, you need to restart BIND:

sudo service bind9 restart

To convert your file to RPZ format, you run:

cat bind | tr -d \"  | awk ' { print $2" CNAME ." } ' > /etc/bind/rpz.db

The script will convert the entries to the following format:

zeus.developershed.com CNAME .
zeusclicks.com CNAME .
zintext.com CNAME .

Add in the options section of named:

response-policy { zone "rpz"; };

Create the declaration of the RPZ zone:

zone "rpz" {
  type master;
  file "/etc/bind/rpz.db";
};

Add to the top of /etc/bind/rpz.db file:

$TTL    604800
@       IN      SOA     localhost. root.localhost. (
                          2         ; Serial
                     604800         ; Refresh
                      86400         ; Retry
                    2419200         ; Expire
                     604800 )       ; Negative Cache TTL

@       IN      NS      your_dns_fqdn.

Deconfigure that DNS file of yours and restart your BIND server. Evidently the RPZ file can be optimised with wildcards and made much shorter, however even without that optimisation now you won't need so much open files.

As for consulting BIND/DNS logs, they are together with the system logs in /var/log/syslog with the tag named. You can use the command:

sudo grep named /var/log/syslog