0

I've just found a bug in an embedded linux machine caused by a broken /etc/resov.conf. Specifically this was a broken symlink. The thing I can't explain is why some (many) of them ever worked.

The code is running inside an Alpine Linux chroot. The Debian "host" has a present and correct resolv.conf, but link iside the chroot environment was broken. I've also discovered that Alpine does not, by default, come with an nsswitch.conf. All hosts have the same Alpine linux chroot environment, they only differ by host version of Debian and network (physical location).

The host and chroot environment share /run /sys /proc /dev only.

Despite this, some devices could resolve eternal FQDNs. I had thought this might be related to shared DNS cache, but have ruled this out. From inside the chroot:

$ ping unix.stackexchange.com
PING unix.stackexchange.com (151.101.193.69): 56 data bytes
64 bytes from 151.101.193.69: seq=0 ttl=58 time=3.962 ms

There is no chance that these embedded devices have been asked to resolve unix.stackexchange.com before so it should not be in their DNS cache.

We are currently investigating to see if the difference between "working" and "broken" devices is the network they are on or subtle differences in release versions. I'll update the question if we can get to the bottom of that detail.

Can anyone explain why some of these devices might be resolving FQDNs without a valid resolv.conf?


Additional details on the chroot

  • All are running the same chroot image.
  • The chroot environment is a "stock" Alpine Mini Root File System with only python3 added via apk add.
  • (Previously mentioned) The chroot file system has no nsswitch.conf and only a very broken link for /etc/resolv.conf

Addational details on the host:

3
  • There's not enough information here to pick one answer. I can think of several possibilities, but to know which is the case one would need to know information that you aren't providing, such as what resolving proxy DNS server actually ended up doing the work, and whether these machines are running their own resolving proxy DNS servers. This involves knowing what is running on the machines, which only you know, and the use of actual DNS diagnostic tools, ping is not such a tool.
    – JdeBP
    Jul 24, 2020 at 16:20
  • How are the systems getting an IP? Via dhcp? Remember that DHCP usually supply the IP of a DNS resolver upstream.
    – user232326
    Jul 24, 2020 at 17:16
  • @JdeBP You say there's "several possibilities". I'd be very grateful if you can name one way a stock Alpine Miniroot File System chroot environment can pick up any DNS settings with no nsswitch.conf and a broken resolv.conf link. Besides the fact that /run is shared with the "host" Debian Linux, I'm really at a loss to know what information to share. Jul 24, 2020 at 20:19

2 Answers 2

3

/etc/resolve.conf is not a mandatory component for resolving hosts. The authoritative interface is the getaddrinfo (or the deprecated gethostbyname) library function, which applications call to resolve host names. Depending on how the system is configured, resolv.conf may or may not be consulted.

For example, when DNS queries are routed via systemd-resolved (configured to do so in /etc/nsswitch.conf), no /etc/resolv.conf is needed. Depending on how systemd-resolved is in turn configured, it can use resolv.conf, or it can maintain /run/systemd/resolve/resolv.conf which may be symlinked from /etc/resolv.conf. See the systemd-resolved man page.

2
  • This might be a good lead. It certainly answers the headline question. I should have mentioned that all the alpine chroot environments are byte-for-byte identical. So I'm struggling to make the connection between different debian hosts and chroot nsswitch.conf. The only one with effect should be the chroot's version and should be read by the chroot libraries right? Jul 24, 2020 at 16:04
  • Now that I've done a bit more research, this answer appears to be a bit misleading for it's example. Alpine linux relies heavily on Busybox and so does not make use nss (for ping). You are right to say that resolv.conf is not mandatory, but that much is self evident from the question. Jul 25, 2020 at 1:31
1

Alpine Linux uses Busybox and uClibc rather than the better known glibc. There's a quirk in uClibc's implementation of getaddrinfo() where it will send DNS requests to localhost:53 when it can find no other name servers in /etc/resolv.conf.

This explains why some units were performing this way and not others despite having identical chroot environments.

It seems dnsmasq was installed on some hosts and not others. Presumably due to the changing decisions of the BeagleBone developers.

The trace for ping -c1 unix.stackexchange.com belo shows Alpine / Busybox / ping attempting to open /etc/resolv.conf and when it fails it sends a request to 127.0.0.1:53. It then get's a response back for that address.

execve("/bin/ping", ["ping", "-c1", "unix.stackexchange.com"], 0xbec18768 /* 20 vars */) = 0
set_tls(0xb6f215ec)                     = 0
set_tid_address(0xb6f221a0)             = 18047
mprotect(0x536000, 8192, PROT_READ)     = 0
getuid32()                              = 0
getpid()                                = 18047
open("/etc/hosts", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
fcntl64(3, F_SETFD, FD_CLOEXEC)         = 0
read(3, "127.0.0.1\tlocalhost localhost.lo"..., 1024) = 79
read(3, "", 1024)                       = 0
close(3)                                = 0
open("/etc/resolv.conf", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
clock_gettime(CLOCK_REALTIME, {tv_sec=1595644595, tv_nsec=840201586}) = 0
clock_gettime(CLOCK_REALTIME, {tv_sec=1595644595, tv_nsec=840992292}) = 0
socket(AF_INET, SOCK_DGRAM|SOCK_CLOEXEC|SOCK_NONBLOCK, IPPROTO_IP) = 3
bind(3, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("0.0.0.0")}, 16) = 0
clock_gettime(CLOCK_REALTIME, {tv_sec=1595644595, tv_nsec=843970581}) = 0
sendto(3, "\247\206\1\0\0\1\0\0\0\0\0\0\4unix\rstackexchange\3"..., 40, MSG_NOSIGNAL, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("127.0.0.1")}, 16) = 40
sendto(3, "\270D\1\0\0\1\0\0\0\0\0\0\4unix\rstackexchange\3"..., 40, MSG_NOSIGNAL, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("127.0.0.1")}, 16) = 40
poll([{fd=3, events=POLLIN}], 1, 2500)  = 1 ([{fd=3, revents=POLLIN}])
recvfrom(3, "\247\206\201\200\0\1\0\4\0\0\0\0\4unix\rstackexchange\3"..., 512, 0, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("127.0.0.1")}, [16]) = 104
recvfrom(3, "\270D\201\200\0\1\0\0\0\1\0\0\4unix\rstackexchange\3"..., 512, 0, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("127.0.0.1")}, [16]) = 124
close(3)                                = 0
ioctl(1, TIOCGWINSZ, {ws_row=38, ws_col=178, ws_xpixel=0, ws_ypixel=0}) = 0
writev(1, [{iov_base="PING unix.stackexchange.com (151"..., iov_len=47}, {iov_base=" data bytes\n", iov_len=12}], 2PING unix.stackexchange.com (151.101.65.69): 56 data bytes
) = 59
socket(AF_INET, SOCK_RAW, IPPROTO_ICMP) = 3
dup2(3, 0)                              = 0
close(3)                                = 0
setsockopt(0, SOL_SOCKET, SO_BROADCAST, [1], 4) = 0
setsockopt(0, SOL_SOCKET, SO_RCVBUF, [7280], 4) = 0
rt_sigprocmask(SIG_UNBLOCK, [RT_1 RT_2], NULL, 8) = 0
rt_sigaction(SIGINT, {sa_handler=0x475b3c, sa_mask=[], sa_flags=SA_RESTORER|SA_RESTART, sa_restorer=0xb6ec89fc}, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
clock_gettime(CLOCK_MONOTONIC, {tv_sec=4280439, tv_nsec=868933133}) = 0
sendto(0, "\10\0Z\r\177F\0\0\5\16\31\236\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 64, 0, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("151.101.65.69")}, 28) = 64
rt_sigaction(SIGALRM, {sa_handler=0x475b3c, sa_mask=[], sa_flags=SA_RESTORER|SA_RESTART, sa_restorer=0xb6ec89fc}, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
setitimer(ITIMER_REAL, {it_interval={tv_sec=0, tv_usec=0}, it_value={tv_sec=10, tv_usec=0}}, {it_interval={tv_sec=0, tv_usec=0}, it_value={tv_sec=0, tv_usec=0}}) = 0
recvfrom(0, "E\0\0T8<\0\0:\1\253\215\227eAE\300\250\3\215\0\0b\r\177F\0\0\5\16\31\236"..., 192, 0, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("151.101.65.69")}, [16]) = 84
clock_gettime(CLOCK_MONOTONIC, {tv_sec=4280439, tv_nsec=879592488}) = 0
writev(1, [{iov_base="64 bytes from 151.101.65.69: seq"..., iov_len=56}, {iov_base="\n", iov_len=1}], 264 bytes from 151.101.65.69: seq=0 ttl=58 time=10.659 ms
) = 57
rt_sigaction(SIGINT, {sa_handler=SIG_IGN, sa_mask=[], sa_flags=SA_RESTORER|SA_RESTART, sa_restorer=0xb6ec89fc}, {sa_handler=0x475b3c, sa_mask=[], sa_flags=SA_RESTORER|SA_RESTART, sa_restorer=0xb6ec89fc}, 8) = 0
writev(1, [{iov_base="", iov_len=0}, {iov_base="\n", iov_len=1}], 2
) = 1
writev(1, [{iov_base="--- unix.stackexchange.com", iov_len=26}, {iov_base=" ping statistics ---\n", iov_len=21}], 2--- unix.stackexchange.com ping statistics ---
) = 47
writev(1, [{iov_base="1 packets transmitted, 1 packets"..., iov_len=45}, {iov_base=" packet loss\n", iov_len=13}], 21 packets transmitted, 1 packets received, 0% packet loss
) = 58
writev(1, [{iov_base="round-trip min/avg/max = 10.659/"..., iov_len=45}, {iov_base=" ms\n", iov_len=4}], 2round-trip min/avg/max = 10.659/10.659/10.659 ms
) = 49
exit_group(0)                           = ?
+++ exited with 0 +++

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .