3

On an embedded Linux device, I am running an application that insists on doing a reverse lookup of the IP address 0.0.0.0 upon startup. This ultimately ends up calling gethostbyaddr.

Sometimes the gethostbyaddr call completes immediately (with failure), but sometimes I see delays (DNS timeouts?) of either 5 or 10 seconds. I have not yet been able to identify in which cases the delays occur.

Some hints:

  • IP6 support is disabled at startup by running this in one of the early init scripts:

    echo 1 >/proc/sys/net/ipv6/conf/all/disable_ipv6
    echo 1 >/proc/sys/net/ipv6/conf/default/disable_ipv6
    echo 1 >/proc/sys/net/ipv6/conf/lo/disable_ipv6
    

    which (I assume) should discard this: https://www.netroby.com/view/3695

  • I am not using Avahi / MDNS, which should discard this: https://bugs.launchpad.net/ubuntu/+source/nss-mdns/+bug/94940

  • The device is not running a local DNS service

  • The problem happens either with DHCP or with a static IP configuration. When using DHCP, my DSL router advertises itself as a DNS server. But when using a static IP configuration, I use 8.8.8.8 as a DNS server (so this should discard a buggy DNS server at the DSL router side)

Any ideas of what may be going on here?


Update:

Relevant line in /etc/nsswitch.conf is currently:

hosts: files dns

I managed to reproduce the problem with a simplified test app. Here's a fragment of the strace output when the timeout occurs:

291   23:34:30 connect(6, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("8.8.8.8")}, 16) = 0 <0.000077>
291   23:34:30 gettimeofday({tv_sec=1514849670, tv_usec=139862}, NULL) = 0 <0.000032>
291   23:34:30 poll([{fd=6, events=POLLOUT}], 1, 0) = 1 ([{fd=6, revents=POLLOUT}]) <0.000049>
291   23:34:30 send(6, "\r\231\1\0\0\1\0\0\0\0\0\0\0010\0010\0010\0010\7in-addr\4arp"..., 38, MSG_NOSIGNAL) = 38 <0.000128>
291   23:34:30 poll([{fd=6, events=POLLIN}], 1, 5000) = 0 (Timeout) <5.005152>
291   23:34:35 gettimeofday({tv_sec=1514849675, tv_usec=147536}, NULL) = 0 <0.000088>
291   23:34:35 poll([{fd=6, events=POLLOUT}], 1, 0) = 1 ([{fd=6, revents=POLLOUT}]) <0.000086>
291   23:34:35 send(6, "\r\231\1\0\0\1\0\0\0\0\0\0\0010\0010\0010\0010\7in-addr\4arp"..., 38, MSG_NOSIGNAL) = 38 <0.000206>
291   23:34:35 poll([{fd=6, events=POLLIN}], 1, 5000) = 1 ([{fd=6, revents=POLLIN}]) <0.045356>
291   23:34:35 ioctl(6, FIONREAD, [106]) = 0 <0.000087>
291   23:34:35 recvfrom(6, "\r\231\201\203\0\1\0\0\0\1\0\0\0010\0010\0010\0010\7in-addr\4arp"..., 1024, 0, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("8.8.8.8")}, [28->16]) = 106 <0.000096>

The first DNS request gets no response, then after a 5 second timeout there is a retry which gets a response almost immediately.

3
  • 1
    What do you have for hosts: in /etc/nsswitch.conf? – derobert Aug 26 '19 at 17:31
  • 1
    Also, can you reproduce with getent hosts 0.0.0.0 ? If so, you can easily strace that to get an idea of what it's waiting on. – derobert Aug 26 '19 at 17:36
  • @derobert Thanks for your comments, I have edited the question to add some extra info. – Grodriguez Aug 27 '19 at 7:43
1

It's possible that the network wasn't fully up (so the DNS request was lost), or maybe it was just a random packet loss (those do happen, especially on WiFi). The former you can deal with by waiting a little longer to start the app; the later is pretty much unavoidable.

Some approaches that might help you, though:

  • Fix the app. I suspect this isn't an option, or you'd have already done it.

  • Put an entry for 0.0.0.0 in /etc/hosts. Then it should resolve through "files", which will be basically instant. Of course, this requires giving 0.0.0.0 a name, which may or may not change app behavior. If this works for you, it's the easiest option.

  • If you don't need DNS on the box, disable it entirely by removing it from /etc/nsswitch.conf.

  • Run a local DNS cache which can be configured to be authoritative for 0.0.0.in-addr.arpa. Then it can quickly return NXDOMAIN for 0.0.0.0. There are several programs intended to provide local caching which can do this. E.g., dnsmasq is reasonably popular. There are others, such as Unbound, depending on the features you need.

  • At least with glibc, you can change the timeout & retry behavior by putting something like options timeout:2 attempts:4 in /etc/resolv.conf. If you decrease the timeout, you probably want to increase the attempts because sometimes it can take the DNS server a while to resolve it (but once it has, it should quickly respond from its cache).

  • You could write your own NSS module to quickly fail hosts lookups for 0.0.0.0. At least for glibc, you can find instructions in the glibc manual §29.

6
  • Great answer. I had already tried to put 0.0.0.0 in /etc/hosts and it does the trick. As you say this is the easiest and it works around the problem. But obviously there is a problem somewhere, and I'd like to fully understand it. I am puzzled by your comment: "It's possible that the network wasn't fully up". Once the kernel reports that the link is up, doesn't that mean that the network is available? If not, wouldn't that be a bug? Or is it normal that the link is reported as up but the network is not yet ready? (if so, is there any way to check for this condition?) – Grodriguez Aug 28 '19 at 7:34
  • BTW this does not seem like a random packet loss -- it happens relatively often on boot. Subsequent DNS requests (once the system is up and running) don't seem to have any problem... – Grodriguez Aug 28 '19 at 7:41
  • @Grodriguez link up happens before, e.g., IP addresses are assigned. Sometimes there are bugs where the firmware reports ready before it really is. Even when everything is working right, sometimes it takes a second for the network (e.g., switches) to notice a new device. Ideally none of this would happen, I guess, but shrug it seems to. – derobert Aug 28 '19 at 8:42
  • OK I think I have some work to do. I should be able to use Wireshark to check if that first DNS request is actually being sent out or not. Thanks for the useful hints, and for getting me on the right track. – Grodriguez Aug 28 '19 at 9:29
  • @Grodriguez and if the response gets back (could be either direction not ready). – derobert Aug 28 '19 at 9:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.