DNS server selection between two LAN interfaces

Question

I have an box running Ubuntu 18.04.4. There are two LAN interfaces enp3s0 and eno1: the former is connected to a 3G modem on LAN network and the latter is connected to a satellite modem on another network. Network Manager takes care of setting priority to the interfaces through Connectivity Check setting in its configuration file (let's say, in case of no 3G coverage eno1 interface metrics is downgraded by adding 20000 to its default value and same applies to enp3s0 in case of no satellite - default priority is higher on eno1 with both connections active). DNS is handled by systemd-resolved and systemd-networked. Both interfaces are getting their IP addresses and DNS from DHCP services run on 3G and satellite modems. enp3s0: IP 192.168.200.101 (DNS/GW: 192.168.200.101) - this address is assigned permanently via MAC address assignment eno1: IP 192.168.55.xxx (DNS/GW: 192.168.55.1)

Somehow DNS server address supplied to the first interface takes priority over the second one. E.g., if enp3s0 does not have internet connection and internet traffic is routed to 192.168.55.1, name resolution is still attempted via 192.168.200.1 and obviously fails. I tried replacing dynamically assigned DNS for eno1 with a static DNS added to /etc/systemd/network/eno1.network

[Match]
Name=eno1

[Network]
DNS=8.8.8.8
DNS=8.8.4.4

But the system still prefers to use 192.168.200.1 for name resolution. I added a global fallback DNS to /etc/systemd/resolved.conf but it does not seem to help if even taken into account.

[Resolve]
#DNS=
FallbackDNS=8.8.4.4
#Domains=
#LLMNR=no
#MulticastDNS=no
#DNSSEC=no
#Cache=yes
#DNSStubListener=yes

What I would like to achieve is to not use only 192.168.200.1 as the DNS server but in case it does not work to use the DNS server associate with the second interface but I am failing to find the way. My understanding it is somehow related to how the first DNS is configured, it got that ~ in its domain assignment what I suspect makes it the main source of internet names but may be I am dreaming. Any advice on that is highly appreciated, basically how can I get both DNSs working? Below is my resolved status output where I see that ~ (with Google DNS assigned to 2nd iface).

Global
          DNSSEC NTA: 10.in-addr.arpa
                      16.172.in-addr.arpa
                      168.192.in-addr.arpa
                      17.172.in-addr.arpa
                      18.172.in-addr.arpa
                      19.172.in-addr.arpa
                      20.172.in-addr.arpa
                      21.172.in-addr.arpa
                      22.172.in-addr.arpa
                      23.172.in-addr.arpa
                      24.172.in-addr.arpa
                      25.172.in-addr.arpa
                      26.172.in-addr.arpa
                      27.172.in-addr.arpa
                      28.172.in-addr.arpa
                      29.172.in-addr.arpa
                      30.172.in-addr.arpa
                      31.172.in-addr.arpa
                      corp
                      d.f.ip6.arpa
                      home
                      internal
                      intranet
                      lan
                      local
                      private
                      test

Link 15 (tun0)
      Current Scopes: none
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no

Link 14 (veth2e5ae19)
      Current Scopes: none
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no

Link 12 (veth5b411fa)
      Current Scopes: none
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no

Link 10 (br-b950c350c024)
      Current Scopes: none
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no

Link 9 (docker0)
      Current Scopes: none
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no

Link 8 (can0)
      Current Scopes: none
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no

Link 7 (wlp1s0)
      Current Scopes: none
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no

Link 6 (wwp0s20u6i10)
      Current Scopes: none
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no

Link 5 (wwp0s20u6i8)
      Current Scopes: none
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no

Link 4 (enp3s0)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: 192.168.200.1
          DNS Domain: ~.
                      rig

Link 3 (enp2s0)
      Current Scopes: none
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no

Link 2 (eno1)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: 8.8.8.8
                      8.8.4.4

Answer 1

In the systemd-resolved terminology, a domain prefixed by ~ in DNS Domain fields indicates "direct queries for this domain into system-wide default DNS server: don't use per-link DNS servers for this domain". The combination ~. is the same, but for the root DNS domain . which is the implied suffix of all DNS domains.

But the problem seems to be that you don't seem to have any system-wide default DNS servers: you have only per-link DNS servers configured. FallbackDNS= is only used if no other DNS server information is known, according to resolved.conf(5) man page. Because both eno1 and enp3s0 have defined per-link DNS servers, FallbackDNS does not get used at all.

Your post says the network configuration for enp3s0 is

enp3s0: IP 192.168.200.101 (DNS/GW: 192.168.200.101)

Because it does not make sense for an interface to be its own gateway (although it's sometimes used as a workaround if there is no connectivity outside the segment and some configuration tool insists on having to configure a gateway), I assume that the DNS/GW part has a typo and you meant "DNS/GW: 192.168.200.1". That would match what the resolved status says.

So if the 3G modem has the IP address 192.168.200.1, then the root cause could be that even if the 3G modem does not have internet connectivity, resolved can still contact the modem itself. So resolved might think the 192.168.200.1 is still a valid DNS server even if the 3G link is down. And if the 3G modem's DNS server/proxy functionality is poorly written, it might not respond with SERVFAIL or by just plain letting the request time out if it does not have a link: this might further mislead resolved into thinking the 3G modem is a valid DNS server when it actually does not have an active link to internet at all.

A DNS proxy in a 3G modem like this might simplify the network configuration when you have just one outgoing internet connection, but it may complicate matters when you have an alternative connection: then you need an

I'd suggest that you specify the static DNS in /etc/systemd/resolved.conf with just DNS= instead of FallbackDNS=. On resolved.conf(5) man page, the description of DNS= says:

DNS=

A space-separated list of IPv4 and IPv6 addresses to use as system DNS servers. DNS requests are sent to one of the listed DNS servers in parallel to suitable per-link DNS servers acquired from systemd-networkd.service(8) or set at runtime by external applications. For compatibility reasons, if this setting is not specified, the DNS servers listed in /etc/resolv.conf are used instead, if that file exists and any servers are configured in it. This setting defaults to the empty list.

So the DNS= setting in resolved.conf will never block the use of per-link DNS servers.

To resolve your issue, I'd suggest the following:

If 192.168.200.1 is the IP address of the 3G modem, it's just acting as a proxy for the 3G network operator's DNS server(s). Find out the real IP addresses of those servers (possibly by checking the internet-side network settings of the 3G modem itself, while it has a link). Then configure those using a DNS= line in /etc/systemd/resolved.conf.

After this, the DNS requests should go in parallel to the 3G network's DNS server(s) (because of a DNS= line in resolved.conf) and whichever per-link DNS server is applicable. Since you now have a system-wide DNS server configured, the 3G modem's built-in DNS service should now only be used if querying for a name in domain *.rig, as that is what the DNS Domain: settings for enp3s0 should mean.

If the 3G network link is down, the attempt to use the 3G network's DNS servers directly should now unquestionably fail (instead of producing possibly-ambiguous responses from the modem itself), giving resolved a clue that this DNS server is no longer available and it should consider other alternatives. As soon as the Connectivity Check has increased the priority of the other link, its per-link DNS servers should start to be used.

If the 3G modem's administration web interface is accessible using a name like <something>.rig, it will still be available using that name, regardless of whether the 3G link is up or down, as long as the modem itself is accessible.