15

I've a master bind9 DNS server and 2 slave servers running on IPv4 (Debian Jessie), using /etc/bind/named.conf:

listen-on-v6 { none; };

When I try to connect from different server(s) each connection takes at least 5 seconds (I'm using Joseph's timing info for debugging):

$ curl -w "@curl-format.txt" -o /dev/null -s https://example.com
            time_namelookup:  5.512
               time_connect:  5.512
            time_appconnect:  5.529
           time_pretransfer:  5.529
              time_redirect:  0.000
         time_starttransfer:  5.531
                            ----------
                 time_total:  5.531

According to curl, lookup takes most of the time, however standard nslookup is very fast:

$ time nslookup example.com > /dev/null 2>&1

real    0m0.018s
user    0m0.016s
sys     0m0.000s

After forcing curl to use IPv4, it gets much better:

$ curl -4 -w "@curl-format.txt" -o /dev/null -s https://example.com

            time_namelookup:  0.004
               time_connect:  0.005
            time_appconnect:  0.020
           time_pretransfer:  0.020
              time_redirect:  0.000
         time_starttransfer:  0.022
                            ----------
                 time_total:  0.022

I've disabled IPv6 on the host:

echo 1 > /proc/sys/net/ipv6/conf/eth0/disable_ipv6

though the problem persists. I've tried running strace to see what's the reason of timeouts:

write(2, "*", 1*)                        = 1
write(2, " ", 1 )                        = 1
write(2, "Hostname was NOT found in DNS ca"..., 36Hostname was NOT found in DNS cache
) = 36
socket(PF_INET6, SOCK_DGRAM, IPPROTO_IP) = 4
close(4)                                = 0
mmap(NULL, 8392704, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7f220bcf8000
mprotect(0x7f220bcf8000, 4096, PROT_NONE) = 0
clone(child_stack=0x7f220c4f7fb0, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tidptr=0x7f220c4f89d0, tls=0x7f220c4f8700, child_tidptr=0x7f220c4f89d0) = 2004
rt_sigaction(SIGPIPE, {SIG_IGN, [PIPE], SA_RESTORER|SA_RESTART, 0x7f22102e08d0}, NULL, 8) = 0
rt_sigaction(SIGPIPE, NULL, {SIG_IGN, [PIPE], SA_RESTORER|SA_RESTART, 0x7f22102e08d0}, 8) = 0
rt_sigaction(SIGPIPE, {SIG_IGN, [PIPE], SA_RESTORER|SA_RESTART, 0x7f22102e08d0}, NULL, 8) = 0
rt_sigaction(SIGPIPE, {SIG_IGN, [PIPE], SA_RESTORER|SA_RESTART, 0x7f22102e08d0}, NULL, 8) = 0
poll(0, 0, 4)                           = 0 (Timeout)
rt_sigaction(SIGPIPE, NULL, {SIG_IGN, [PIPE], SA_RESTORER|SA_RESTART, 0x7f22102e08d0}, 8) = 0
rt_sigaction(SIGPIPE, {SIG_IGN, [PIPE], SA_RESTORER|SA_RESTART, 0x7f22102e08d0}, NULL, 8) = 0
rt_sigaction(SIGPIPE, {SIG_IGN, [PIPE], SA_RESTORER|SA_RESTART, 0x7f22102e08d0}, NULL, 8) = 0
poll(0, 0, 8)                           = 0 (Timeout)
rt_sigaction(SIGPIPE, NULL, {SIG_IGN, [PIPE], SA_RESTORER|SA_RESTART, 0x7f22102e08d0}, 8) = 0
rt_sigaction(SIGPIPE, {SIG_IGN, [PIPE], SA_RESTORER|SA_RESTART, 0x7f22102e08d0}, NULL, 8) = 0
rt_sigaction(SIGPIPE, {SIG_IGN, [PIPE], SA_RESTORER|SA_RESTART, 0x7f22102e08d0}, NULL, 8) = 0
poll(0, 0, 16)                          = 0 (Timeout)
rt_sigaction(SIGPIPE, NULL, {SIG_IGN, [PIPE], SA_RESTORER|SA_RESTART, 0x7f22102e08d0}, 8) = 0
rt_sigaction(SIGPIPE, {SIG_IGN, [PIPE], SA_RESTORER|SA_RESTART, 0x7f22102e08d0}, NULL, 8) = 0
rt_sigaction(SIGPIPE, {SIG_IGN, [PIPE], SA_RESTORER|SA_RESTART, 0x7f22102e08d0}, NULL, 8) = 0
poll(0, 0, 32)                          = 0 (Timeout)
rt_sigaction(SIGPIPE, NULL, {SIG_IGN, [PIPE], SA_RESTORER|SA_RESTART, 0x7f22102e08d0}, 8) = 0
rt_sigaction(SIGPIPE, {SIG_IGN, [PIPE], SA_RESTORER|SA_RESTART, 0x7f22102e08d0}, NULL, 8) = 0
rt_sigaction(SIGPIPE, {SIG_IGN, [PIPE], SA_RESTORER|SA_RESTART, 0x7f22102e08d0}, NULL, 8) = 0
poll(0, 0, 64)                          = 0 (Timeout)

It doesn't seem to be a firewall issues as nslookup (or curl -4) is using the same DNS servers. Any idea what could be wrong?

Here's tcpdump from the host tcpdump -vvv -s 0 -l -n port 53:

tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
20:14:52.542526 IP (tos 0x0, ttl 64, id 35839, offset 0, flags [DF], proto UDP (17), length 63)
    192.168.1.1.59163 > 192.168.1.2.53: [bad udp cksum 0xf9f3 -> 0x96c7!] 39535+ A? example.com. (35)
20:14:52.542540 IP (tos 0x0, ttl 64, id 35840, offset 0, flags [DF], proto UDP (17), length 63)
    192.168.1.1.59163 > 192.168.1.2.53: [bad udp cksum 0xf9f3 -> 0x6289!] 45997+ AAAA? example.com. (35)
20:14:52.543281 IP (tos 0x0, ttl 61, id 63674, offset 0, flags [none], proto UDP (17), length 158)
    192.168.1.2.53 > 192.168.1.1.59163: [udp sum ok] 45997* q: AAAA? example.com. 1/1/0 example.com. [1h] CNAME s01.example.com. ns: example.com. [10m] SOA ns01.example.com. ns51.domaincontrol.com. 2016062008 28800 7200 1209600 600 (130)
20:14:57.547439 IP (tos 0x0, ttl 64, id 36868, offset 0, flags [DF], proto UDP (17), length 63)
    192.168.1.1.59163 > 192.168.1.2.53: [bad udp cksum 0xf9f3 -> 0x96c7!] 39535+ A? example.com. (35)
20:14:57.548188 IP (tos 0x0, ttl 61, id 64567, offset 0, flags [none], proto UDP (17), length 184)
    192.168.1.2.53 > 192.168.1.1.59163: [udp sum ok] 39535* q: A? example.com. 2/2/2 example.com. [1h] CNAME s01.example.com., s01.example.com. [1h] A 136.243.154.168 ns: example.com. [30m] NS ns01.example.com., example.com. [30m] NS ns02.example.com. ar: ns01.example.com. [1h] A 136.243.154.168, ns02.example.com. [1h] A 192.168.1.2 (156)
20:14:57.548250 IP (tos 0x0, ttl 64, id 36869, offset 0, flags [DF], proto UDP (17), length 63)
    192.168.1.1.59163 > 192.168.1.2.53: [bad udp cksum 0xf9f3 -> 0x6289!] 45997+ AAAA? example.com. (35)
20:14:57.548934 IP (tos 0x0, ttl 61, id 64568, offset 0, flags [none], proto UDP (17), length 158)
    192.168.1.2.53 > 192.168.1.1.59163: [udp sum ok] 45997* q: AAAA? example.com. 1/1/0 example.com. [1h] CNAME s01.example.com. ns: example.com. [10m] SOA ns01.example.com. ns51.domaincontrol.com. 2016062008 28800 7200 1209600 600 (130)

EDIT: In bind logs frequently appears this message:

error sending response: host unreachable

Though, each query is eventually answered (it just takes 5s). All machines are physical servers (it's not fault of NAT), it's more likely that packets are being blocked by a router. Here's quite likely related question: DNS lookups sometimes take 5 seconds.

Improve this question
3
  • 1
    strace -tt would make the trace more informative when tracking down delays.
    – JigglyNaga
    Commented Jun 20, 2016 at 22:03
  • Thanks, it doesn't help much in this case. It seems to be stuck in a look while retrying same connection with increasing timeout poll(0, 0, 1000) = 0 (Timeout). On the DNS server side I'm getting frequent errors error sending response: host unreachable which looks like the outgoing packet is blocked (but not for nslookup).
    – Tombart
    Commented Jun 20, 2016 at 22:26
  • This looks similar- philippecloutier.com/…
    – Jeff Schaller ♦
    Commented Jun 20, 2016 at 23:41

5 Answers 5

Reset to default
18

Short answer:

A workaround is forcing glibc to reuse a socket for look up of the AAAA and A records, by adding a line to /etc/resolv.conf:

options single-request-reopen

The real cause of this issue might be:

Long answer:

Programs like curl or wget use glibc's function getaddrinfo(), which tries to be compatible with both IPv4 and IPv6 by looking up both DNS records in parallel. It doesn't return result until both records are received (there are several issues related to such behaviour) - this explains the strace above. When IPv4 is forced, like curl -4 internally gethostbyname() which queries for A record only.

From tcpdump we can see that:

  • -> A? two requests are send at the beginning
  • -> AAAA? (requesting IPv6 address)
  • <- AAAA reply
  • -> A? requesting again IPv4 address
  • <- A got reply
  • -> AAAA? requesting IPv6 again
  • <- AAAA reply

One A reply gets dropped for some reason, that's this error message:

error sending response: host unreachable

Yet it's unclear to me why there's a need for second AAAA query.

To verify that you're having the same issue you can update timeout in /etc/resolv.conf:

options timeout:3

First create a text file with custom time reporting config:

cat >./curl-format.txt  <<-EOF
   time_namelookup: %{time_namelookup}\n
      time_connect: %{time_connect}\n
   time_appconnect: %{time_appconnect}\n
     time_redirect: %{time_redirect}\n
  time_pretransfer: %{time_pretransfer}\n
time_starttransfer: %{time_starttransfer}\n
                    ----------\n
time_total: %{time_total}\n
EOF

then send a request:

$ curl -w "@curl-format.txt" -o /dev/null -s https://example.com

            time_namelookup:  3.511
               time_connect:  3.511
            time_appconnect:  3.528
           time_pretransfer:  3.528
              time_redirect:  0.000
         time_starttransfer:  3.531
                            ----------
                 time_total:  3.531

There are two other related options in man resolv.conf:

single-request (since glibc 2.10) sets RES_SNGLKUP in _res.options. By default, glibc performs IPv4 and IPv6 lookups in parallel since version 2.9. Some appliance DNS servers cannot handle these queries properly and make the requests time out. This option disables the behavior and makes glibc perform the IPv6 and IPv4 requests sequentially (at the cost of some slowdown of the resolving process).

single-request-reopen (since glibc 2.9) The resolver uses the same socket for the A and AAAA requests. Some hardware mistakenly sends back only one reply. When that happens the client system will sit and wait for the second reply. Turning this option on changes this behavior so that if two requests from the same port are not handled correctly it will close the socket and open a new one before sending the second request.

Related issues:

Improve this answer
1
  • @RuiFRibeiro Solely based on the up-voltes it seemed that users found the other answer more helpful. No offense meant.
    – Tombart
    Commented Oct 10, 2018 at 19:32
5

As @Tombart says, the delay is due to waiting for the IPv6 resolution timeout.

Another possible course of action is giving precedence to IPv4 in /etc/gai.conf

From comments in /etc/gai.conf

#   For sites which prefer IPv4 connections change the last line to
#
precedence ::ffff:0:0/96  100

After changing gai.conf, you need to restart any app using the DNS resolver library for the change to take effect.

Mind you that if you are using a BIND server without IPv6 connectivity, I advise disabling IPv6 in named and taking from the root hints IPv6 addresses. Obviously it will still try to resolve AAAA addresses.

So for the BIND configuration,

In /etc/default/bind9, add -4 for IPv4 addresses:

OPTIONS="-4 -u bind"

and in /etc/bind/db.root, delete all lines with AAAA DNS roots.

Improve this answer
2

I had a similar issue while using BIND9. To fix this I needed to add:

filter-aaaa-on-v4 yes;

option to my named.conf.

(More information)

Improve this answer
1

This issue was resolved for me after starting to use systemd-resolved. See https://wiki.archlinux.org/title/systemd-resolved

Improve this answer
2
  • Welcome to the site, and thank you for your contribution. Please note that [link-only answers](unix.stackex change.com/help/how-to-answer) are discouraged, as the link may become invalid or the linked content change. Please edit your answer to include at least a summary of the linked explanation.
    – AdminBee
    Commented Oct 31, 2023 at 10:57
  • The link is just optional. The answer is to use systemd-resolved. The installing process depends on the Linux distribution in use.
    – PSanetra
    Commented Nov 1, 2023 at 22:53
0

In case someone is looking for the curl-format.txt. Paste this into your shell and it will create the format file for you. The original link didn't work for me. found this example here

cat >./curl-format.txt  <<-EOF
   time_namelookup: %{time_namelookup}\n
      time_connect: %{time_connect}\n
   time_appconnect: %{time_appconnect}\n
     time_redirect: %{time_redirect}\n
  time_pretransfer: %{time_pretransfer}\n
time_starttransfer: %{time_starttransfer}\n
                    ----------\n
time_total: %{time_total}\n
EOF
Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .