2

I'm having problems with SystemD's resolved service on Ubuntu 18.10 - on my wireless network we have an IPv6 subnet that is tunneled through some weird network setup (work stuff) and can't be used for DNS resolution.

For some reason, even though I have set up static DNS configuration in /etc/systemd/resolved.conf.d/dns.conf to only include IPv4 DNS records, and the DHCP never advertises IPv6 DNS servers, resolved always adds some funky IPv6 addresses to the connection-specific DNS configuration on my wireless connection, and then "fallsback" to them and stops resolving addresses.

This is my dns.conf (the primary resolved.conf is default - there's nothing there):

[Resolve]
DNS=172.18.1.254 8.8.8.8 8.8.4.4 1.1.1.1
DNSSEC=no

(the first DNS address is the local router)

The resolved global configuration (as can be gotten from resolvectl status):

Global
       LLMNR setting: no
MulticastDNS setting: no
  DNSOverTLS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
  Current DNS Server: 172.18.1.254
         DNS Servers: 172.18.1.254
                      8.8.8.8
                      8.8.4.4
                      1.1.1.1
...

Link 2 (wlp2s0)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
  DNSOverTLS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
  Current DNS Server: 2001:4860:4860::4444
         DNS Servers: 172.18.1.254
                      8.8.8.8
                      8.8.4.4
                      2001:470:1f09:114c::1
                      2001:4860:4860::4444
                      2001:4860:4860::8888
          DNS Domain: ~.

The wlp2s0 status starts with the current DNS server set as 172.18.1.254, then after a short time (we're talking minutes here), it falls back through all the IPv4 servers and lands on one of the IPv6 servers - after which all DNS queries fail, and I'm basically get disconnected from any meaningful network operation. A systemctl restart systemd-resolved clears the issue for a few minutes, after which its back to "no DNS for you, come back when you have a non-IPv6 network".

I have no idea where those IPv6 addresses come from. I've enabled debug logging on resolved, and looking at the debug log the first mention of an IPv6 address is something like this:

Feb 12 10:36:04 vesho systemd-resolved[22371]: Processing incoming packet on transaction 51369 (rcode=SERVFAIL).
Feb 12 10:36:04 vesho systemd-resolved[22371]: Retrying transaction 51369.
Feb 12 10:36:04 vesho systemd-resolved[22371]: Switching to DNS server 2001:470:1f09:114c::1 for interface wlp2s0.
Feb 12 10:36:04 vesho systemd-resolved[22371]: Cache miss for some.domain IN A
Feb 12 10:36:04 vesho systemd-resolved[22371]: Transaction 51369 for <some.domain IN A> scope dns on wlp2s0/*.
Feb 12 10:36:04 vesho systemd-resolved[22371]: Using feature level UDP+EDNS0 for transaction 51369.
Feb 12 10:36:04 vesho systemd-resolved[22371]: Using DNS server 2001:470:1f09:114c::1 for transaction 51369.
Feb 12 10:36:04 vesho systemd-resolved[22371]: Sending query packet with id 51369.

It looks like if something on my system queries an address that cannot be resolved, and the upstream DNS server returns SERVFAIL (this is not just my local router's DNS resolver that is broken - 8.8.8.8 returns the same response), then resolved falls back through all the server until it finds a server that doesn't respond at all - on the way causing any query, not just the one that is broken, to use a broken server.

I can trace the path of such an escalation in the debug log (some.domain is a placeholder for another address that cannot be resolved):

Feb 12 10:36:01 vesho systemd-resolved[22371]: Cache miss for some.domain IN A
Feb 12 10:36:01 vesho systemd-resolved[22371]: Transaction 51369 for <some.domain IN A> scope dns on wlp2s0/*.
Feb 12 10:36:01 vesho systemd-resolved[22371]: Using feature level UDP for transaction 51369.
Feb 12 10:36:01 vesho systemd-resolved[22371]: Using DNS server 172.18.1.254 for transaction 51369.
Feb 12 10:36:01 vesho systemd-resolved[22371]: Sending query packet with id 51369.
Feb 12 10:36:01 vesho systemd-resolved[22371]: Cache miss for some.domain IN AAAA
Feb 12 10:36:01 vesho systemd-resolved[22371]: Transaction 44627 for <some.domain IN AAAA> scope dns on wlp2s0/*.
...
Feb 12 10:36:03 vesho systemd-resolved[22371]: Processing incoming packet on transaction 51369 (rcode=SERVFAIL).
Feb 12 10:36:03 vesho systemd-resolved[22371]: Retrying transaction 51369.
Feb 12 10:36:03 vesho systemd-resolved[22371]: Switching to DNS server 8.8.8.8 for interface wlp2s0.
Feb 12 10:36:03 vesho systemd-resolved[22371]: Cache miss for some.domain IN A
Feb 12 10:36:03 vesho systemd-resolved[22371]: Transaction 51369 for <some.domain IN A> scope dns on wlp2s0/*.
Feb 12 10:36:03 vesho systemd-resolved[22371]: Using feature level UDP+EDNS0 for transaction 51369.
Feb 12 10:36:03 vesho systemd-resolved[22371]: Using DNS server 8.8.8.8 for transaction 51369.
Feb 12 10:36:03 vesho systemd-resolved[22371]: Sending query packet with id 51369.
...
Feb 12 10:36:03 vesho systemd-resolved[22371]: Processing incoming packet on transaction 51369 (rcode=SERVFAIL).
Feb 12 10:36:03 vesho systemd-resolved[22371]: Server returned error SERVFAIL, retrying transaction with reduced feature level UDP.
Feb 12 10:36:03 vesho systemd-resolved[22371]: Retrying transaction 51369.
Feb 12 10:36:03 vesho systemd-resolved[22371]: Cache miss for some.domain IN A
Feb 12 10:36:03 vesho systemd-resolved[22371]: Transaction 51369 for <some.domain IN A> scope dns on wlp2s0/*.
Feb 12 10:36:03 vesho systemd-resolved[22371]: Using feature level UDP for transaction 51369.
Feb 12 10:36:03 vesho systemd-resolved[22371]: Sending query packet with id 51369.
Feb 12 10:36:03 vesho systemd-resolved[22371]: Processing incoming packet on transaction 51369 (rcode=SERVFAIL).
Feb 12 10:36:03 vesho systemd-resolved[22371]: Retrying transaction 51369.
Feb 12 10:36:03 vesho systemd-resolved[22371]: Switching to DNS server 8.8.4.4 for interface wlp2s0.
Feb 12 10:36:03 vesho systemd-resolved[22371]: Cache miss for some.domain IN A
Feb 12 10:36:03 vesho systemd-resolved[22371]: Transaction 51369 for <some.domain IN A> scope dns on wlp2s0/*.
Feb 12 10:36:03 vesho systemd-resolved[22371]: Using feature level UDP+EDNS0 for transaction 51369.
Feb 12 10:36:03 vesho systemd-resolved[22371]: Using DNS server 8.8.4.4 for transaction 51369.
Feb 12 10:36:03 vesho systemd-resolved[22371]: Sending query packet with id 51369.
Feb 12 10:36:04 vesho systemd-resolved[22371]: Processing incoming packet on transaction 51369 (rcode=SERVFAIL).
Feb 12 10:36:04 vesho systemd-resolved[22371]: Server returned error SERVFAIL, retrying transaction with reduced feature level UDP.
Feb 12 10:36:04 vesho systemd-resolved[22371]: Retrying transaction 51369.
Feb 12 10:36:04 vesho systemd-resolved[22371]: Cache miss for some.domain IN A
Feb 12 10:36:04 vesho systemd-resolved[22371]: Transaction 51369 for <some.domain IN A> scope dns on wlp2s0/*.
Feb 12 10:36:04 vesho systemd-resolved[22371]: Using feature level UDP for transaction 51369.
Feb 12 10:36:04 vesho systemd-resolved[22371]: Sending query packet with id 51369.
Feb 12 10:36:04 vesho systemd-resolved[22371]: Processing incoming packet on transaction 51369 (rcode=SERVFAIL).
Feb 12 10:36:04 vesho systemd-resolved[22371]: Retrying transaction 51369.
Feb 12 10:36:04 vesho systemd-resolved[22371]: Switching to DNS server 2001:470:1f09:114c::1 for interface wlp2s0.
Feb 12 10:36:04 vesho systemd-resolved[22371]: Cache miss for some.domain IN A
Feb 12 10:36:04 vesho systemd-resolved[22371]: Transaction 51369 for <some.domain IN A> scope dns on wlp2s0/*.
Feb 12 10:36:04 vesho systemd-resolved[22371]: Using feature level UDP+EDNS0 for transaction 51369.
Feb 12 10:36:04 vesho systemd-resolved[22371]: Using DNS server 2001:470:1f09:114c::1 for transaction 51369.
Feb 12 10:36:04 vesho systemd-resolved[22371]: Sending query packet with id 51369.
...
Feb 12 10:36:08 vesho systemd-resolved[22371]: Processing incoming packet on transaction 51369 (rcode=SERVFAIL).
Feb 12 10:36:08 vesho systemd-resolved[22371]: Retrying transaction 51369.
Feb 12 10:36:08 vesho systemd-resolved[22371]: Switching to DNS server 2001:4860:4860::8888 for interface wlp2s0.
Feb 12 10:36:08 vesho systemd-resolved[22371]: Cache miss for some.domain IN A
Feb 12 10:36:08 vesho systemd-resolved[22371]: Transaction 51369 for <some.domain IN A> scope dns on wlp2s0/*.
Feb 12 10:36:08 vesho systemd-resolved[22371]: Using feature level UDP+EDNS0 for transaction 51369.
Feb 12 10:36:08 vesho systemd-resolved[22371]: Using DNS server 2001:4860:4860::8888 for transaction 51369.
Feb 12 10:36:08 vesho systemd-resolved[22371]: Sending query packet with id 51369.
...
Feb 12 10:36:13 vesho systemd-resolved[22371]: Timeout reached on transaction 51369.
Feb 12 10:36:13 vesho systemd-resolved[22371]: Retrying transaction 51369.
Feb 12 10:36:13 vesho systemd-resolved[22371]: Switching to DNS server 2001:4860:4860::8888 for interface wlp2s0.
Feb 12 10:36:13 vesho systemd-resolved[22371]: Cache miss for some.domain IN A
Feb 12 10:36:13 vesho systemd-resolved[22371]: Transaction 51369 for <some.domain IN A> scope dns on wlp2s0/*.
Feb 12 10:36:13 vesho systemd-resolved[22371]: Using feature level UDP+EDNS0 for transaction 51369.
Feb 12 10:36:13 vesho systemd-resolved[22371]: Sending query packet with id 51369.
...
Feb 12 10:36:18 vesho systemd-resolved[22371]: Timeout reached on transaction 51369.
Feb 12 10:36:18 vesho systemd-resolved[22371]: Retrying transaction 51369.
Feb 12 10:36:18 vesho systemd-resolved[22371]: Switching to DNS server 2001:4860:4860::8888 for interface wlp2s0.
Feb 12 10:36:18 vesho systemd-resolved[22371]: Cache miss for some.domain IN A
Feb 12 10:36:18 vesho systemd-resolved[22371]: Transaction 51369 for <some.domain IN A> scope dns on wlp2s0/*.
Feb 12 10:36:18 vesho systemd-resolved[22371]: Using feature level UDP+EDNS0 for transaction 51369.
Feb 12 10:36:18 vesho systemd-resolved[22371]: Sending query packet with id 51369.
... 
goes like that for a while
...
Feb 12 10:36:28 vesho systemd-resolved[22371]: Processing incoming packet on transaction 51369 (rcode=SERVFAIL).
Feb 12 10:36:28 vesho systemd-resolved[22371]: Server returned error: SERVFAIL
Feb 12 10:36:28 vesho systemd-resolved[22371]: Added SERVFAIL cache entry for some.domain IN A 30s
Feb 12 10:36:28 vesho systemd-resolved[22371]: Transaction 51369 for <some.domain IN A> on scope dns on wlp2s0/* now complete with <rcode-failure> from network (unsigned)
Feb 12 10:36:28 vesho systemd-resolved[22371]: Freeing transaction 51369.

So to conclude, my question is: Where does resovled find those IPv6 server addresses and how can I stop it from using them at all (setting a hardcoded DNS or FallbackDNS in the configuration files doesn't seem to stop it)?

If anyone has any suggestion for another solution to this problem, I'd love to hear it. Except if its "disable resolved" - I'm actually doing that now so I can get some work done, but I'd like to get to the bottom of this resolved problem and fix it.

  • The proper IPV6 addresses of Google Public DNS do not, really, qualify as "funky". (-: – JdeBP Feb 13 at 16:59
  • I thought its something like this, but term "funky" here refers to how they appear out of thin air - I did not configure this. – Guss Feb 13 at 17:01
  • Why is systemd-resolved falling back to using Google Public DNS over IPv6 when I did not configure it to do that? has more of the right keywords. (-: – JdeBP Feb 13 at 18:00
  • Thank you for the suggestion, I updated the question subject. – Guss Feb 13 at 18:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.