7

I have configured my DHCP server to supply two nameservers for redundancy, so that if one is offline the other one can be used.

I have configured my PCs with systemd-resolved and according to resolvectl status it has picked up all the DNS servers (IPv4 and IPv6 addresses for both) and is using one as the current one.

However, if the DNS server goes offline, systemd-resolved does not switch to the next server but instead keeps trying to connect to the offline one, causing all uncached name resolution to fail.

If I run systemctl restart systemd-resolved then it will switch to another server and continue working, but it will randomly switch back to the offline server after a while and name resolution will again fail.

How can I tell systemd-resolved to stop using an offline DNS server and quickly switch to one of the other ones it knows about?

journalctl only shows this when it switches to using the offline server:

systemd-resolved[1985]: Using degraded feature set (UDP) for DNS server fdxx::x.
systemd-resolved[1985]: Using degraded feature set (TCP) for DNS server fdxx::x.

The server in question is completely offline when this happens and does not respond to pings.

1
7

DNS professionals have known for long that if you want DNS service resiliency in any network, you do not leave that decision up to the client implementation.

It is a too important decision, to leave to the resolver implementations of clients to make.

While theoretically the clients should fall back to the 2nd DNS upon the failure of the 1st, often that does not happen for multiple reasons. Over the years in my career, I have seen huge failures in complete networks, over people implementing things counting on the client operating system resolver being smart enough.

What you usually do is actually, what the root nameservers are doing, which is to create DNS clusters to take over virtual IP addresses of your DNS servers. The most used technology for that is DNS anycast. You can try also simpler architectures, like using keepdalived.

However, whatever you do, I stress not leaving that decision up to the client.

see IPv4 Anycast with Linux and Quagga

The traditional safeguard is to establish multiple DNS servers for a given site. Each DNS client on the network is configured with each of those servers' IP addresses. The chances of all of those servers failing in a catastrophic way are fairly small, so you have a margin of safety.

On the other hand, many stub resolvers will take only two DNS servers, making it nearly impossible to have any meaningful geographical dispersion in your DNS topology. DNS stub resolvers generally use the first of two configured DNS servers exclusively. Consequently, you end up with one server taking the entire query load and one idling, waiting for a failure. Not optimal, but hey, that's the price of redundancy...right? It doesn't have to be.

DNS redundancy and failover is a classic use case for anycast. Anycast is the concept of taking one IP address and sharing it between multiple servers, each unaware of the others. The DNS root nameservers make extensive use of anycast.

PS. I implemented anycast DNS with iBGP and OSPF, at two ISPs and one University in the past, with dramatic improvements in the uptime availability of the DNS service.

1
  • 2
    Worth an upvote. E.g. regardless of how you can configure systemd-resolved, if you're starting to use a freshly installed laptop which you have not configured systemd-resolved on, you can hit the systemd-resolved DNS problem, or the problem that slm linked to, or just weird long timeouts that delay every single DNS request, while you're trying to check docs on the Web (like this QA page or something :).
    – sourcejedi
    Sep 2 '18 at 8:51
1

Using multiple nameservers via DHCP is for resiliency, not redundancy. It appears you're attempting to use multiple nameservers in the sense that each client will track if a nameserver becomes unresponsive, and stop using it. If you truly want that type of behavior/design then you need to utilize that through a DNS server itself, DNS clients are typically not able to accomplish this. The approach you'll often see here is to make your DNS server HA (highly available), with respect to the upstream DNS servers it's resolving against.

To show why this won't work, take a look at how the /etc/resolv.conf file functions. Adding multiple nameservers to DHCP will give each client 2 entries in their /etc/resolv.conf files. This file can only provide the following mechanism for handling multiple nameservers:

man resolv.conf
  nameserver Name server IP address

          Internet address of a name server that the resolver should query, either 
          an IPv4 address (in dot notation), or an IPv6 address in colon (and 
          possibly dot) notation as per RFC  2373. Up  to MAXNS (currently 3, see 
          <resolv.h>) name servers may be listed, one per keyword.  If there are 
          multiple servers, the resolver library queries them in the order listed.  
          If no nameserver entries are present, the default is to use the name 
          server on the local machine.  (The algorithm used is to try a name server, 
          and if the query times out, try  the  next, until out of name servers, 
          then repeat trying all the name servers until a maximum number of retries 
          are made.)

This sentence states it:

If there are multiple servers, the resolver library queries them in the order listed.

The resolver doesn't do anything to manage this list, it blindly will continue using the list, starting over each time with the 1st entry then the 2nd etc. The only things you can do to control it is change the timeout, retries and rotate.

timeout:n

sets the amount of time the resolver will wait for a response from a remote name server before retrying the query via a different name server. Measured in
seconds, the default is RES_TIMEOUT (currently 5, see ). The value for this option is silently capped to 30.

attempts:n

sets the number of times the resolver will send a query to its name servers before giving up and returning an error to the calling application. The default is RES_DFLRETRY (currently 2, see ). The value for this option is silently capped to 5.

rotate

sets RES_ROTATE in _res.options, which causes round-robin selection of nameservers from among those listed. This has the effect of spreading the query load among all
listed servers, rather than having all clients try the first listed server first every time.

References

13
  • "only achieve" is wrong. the resolv.conf fallback mechanism is not used by systemd-resolved. I've had this when using unbound instead of systemd-resolved, I think it behaved better than in this question, although I am not certain.
    – sourcejedi
    Sep 2 '18 at 8:54
  • If it's not supposed to be used for redundancy, what is it that you are saying you gain in "resiliency?" I don't understand.
    – sourcejedi
    Sep 2 '18 at 9:00
  • @sourcejedi - freedesktop.org/software/systemd/man/….
    – slm
    Sep 2 '18 at 12:08
  • 1
    well yes, you can configure your system to run systemd-resolved but then have your apps use a resolv.conf that does not point to systemd-resolved, but that would be a very strange assumption to make in your answer.
    – sourcejedi
    Sep 2 '18 at 12:18
  • @sourcejedi the behavior he's seeking with his apps failing over to the 2nd ns is resolvers behavior, nothing with systemd's implementation of systems-resolved
    – slm
    Sep 2 '18 at 12:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.