How to avoid conflicts between dnsmasq and systemd-resolved?

Question

I recently installed dnsmasq to act as DNS Server for my local network. dnsmasq listens on port 53 which is already in use by the local DNS stub listener from systemd-resolved.

Just stopping systemd-resolved and then restart it after dnsmasq is running solves this issue. But it returns after a reboot: systemd-resolved is started with preference and dnsmasq will not start because port 53 is already in use.

The first obvious question, I guess, is how do I best make systemd-resolved understand that it should not start the local DNS stub listener and thus keep port 53 for use by dnsmasq?

A more interesting question, however, is how the two services are generally meant to work together. Are they even meant to work side by side or is systemd-resolved just in the way if one's using dnsmasq?

Answer 1

As of systemd 232 (released in 2017) you can edit /etc/systemd/resolved.conf (not /etc/resolv.conf) and add this line:

DNSStubListener=no

This will switch off binding to port 53.

The option is described in more details in the resolved.conf manpage.

You can find the systemd version your system is running with:

systemctl --version

Answer 2

I just enabled option "bind-interfaces" by removing '#' at the start of the line in /etc/dnsmasq.conf.

I was able to start dnsmasq again:

• dnsmasq bind DNS port on all interfaces (including 127.0.0.1) port 53,
• systemd-resolv keeps listening on 127.0.0.53:53

I was pointed to this solution by this discussion resolved: add an option to disable the stub resolver

Answer 3

You can disable systemd-resolved from loading at boot using sudo systemctl disable systemd-resolved.

If you want to run the two together, you can redirect the systemd-resolved to use the localhost as the primary nameserver. This will make sure that all queries are directed to dnsmasq for resolution before hitting the external DNS server. This can be done by adding the line nameserver 127.0.0.1 at the top of your /etc/resolv.conf file. This will also disable systemd's local caching.

You can read more on the Arch Linux wiki. I copied this from there and it covers it quite well.

However this does not reliably avoid the error at boot time, i.e. dnsmasq will still fail if systemd-resolved happens to start first. If your version of systemd is new enough, use the answer by Malvineous. If your version of systemd is too old, you can work around this problem by modifying the dnsmasq unit: in the [Unit] section, add Before=systemd-resolved .

After this, if you like, you can create a separate /etc/dnsmasq-resolv.conf file for the upstream nameservers and pass it using the -r or --resolv-file option, or add the upstream nameservers to the dnsmasq configuration file and use the -R or --no-resolv option. This way you only have the localhost in your /etc/resolv.conf and everything goes through dnsmasq.

Answer 4

Here is solution for (X)Ubuntu 18.04 Bionic.

Install dnsmasq

sudo apt install dnsmasq

Disable systemd-resolved listener on port 53 (do not touch /etc/systemd/resolved.conf, because it may be overwritten on upgrade):

$ cat /etc/systemd/resolved.conf.d/noresolved.conf 
[Resolve]
DNSStubListener=no

and restart it

$ sudo systemctl restart systemd-resolved

(alternatively disable it completely by $ sudo systemctl disable systemd-resolved.service )

Delete /etc/resolv.conf and create again. This is important, because resolv.conf is a symbolic link to /run/systemd/resolve/stub-resolv.conf by default. If you will not delete symbolic link, the file will be overwritten by systemd on reboot (even though we disabled systemd-resolved!). Also NetworkManager (NM) checks if it is a symbolic link to detect systemd-resolved configuration.

$ sudo rm /etc/resolv.conf
$ sudo touch /etc/resolv.conf

Disable overwriting of /etc/resolv.conf by NM (there is also an option rc-manager, but it does not work, despite it is described in the NM manual):

$ cat /etc/NetworkManager/conf.d/disableresolv.conf 
[main]
dns=none

and restart it:

$ sudo systemctl restart NetworkManager

Tell dnsmasq to use resolv.conf from NM:

$ cat /etc/dnsmasq.d/nmresolv.conf 
resolv-file=/var/run/NetworkManager/resolv.conf

and restart it:

$ sudo systemctl restart dnsmasq

Use dnsmasq for resolving:

$ cat /etc/resolv.conf 
# Use local dnsmasq for resolving
nameserver 127.0.0.1

Answer 5

Judging from the systemd manpages it's not intended being able to manually disable the stub DNS server. Interestingly I only noticed the described problem after upgrading systemd from 230 to 231.

Disabling systemd-resolved was no option for me because I need it to handle received upstream DNS servers through DHCP.

My solution was making dnsmasq stop systemd-resolved before starting and starting it afterwards again.

I created a drop-in config in /etc/systemd/system/dnsmasq.service.d/resolved-fix.conf:

[Unit]
After=systemd-resolved.service

[Service]
ExecStartPre=/usr/bin/systemctl stop systemd-resolved.service
ExecStartPost=/usr/bin/systemctl start systemd-resolved.service

This appears to be a rather hackish solution but it works.

Answer 6

There will be an option in systemd version 232 to disable the stub listener. See https://github.com/systemd/systemd/pull/4061.

Answer 7

If you're using a default Ubuntu 18.04 setup, this may be caused by a conflict between systemd-resolved (the default DNS server) and dnsmasq. If you installed dnsmasq yourself deliberately because you explicitly wanted it, then one of the other answers to this question, explaining how to disable systemd-resolved, will probably be good for you. If you didn't explicitly install dnsmasq, then it is likely in place because you're using lxd. This might be because you actually use lxd to manage containers, but it's most likely to be because snaps use lxd to protect you when apps are installed. From my perspective, I want to keep dnsmasq (because lxd wants it) but I also want to keep systemd-resolved as the DNS server (because it's what the Ubuntu team chose and I trust them more than myself).

So, this seems to be an lxd problem at heart. If so, the way I fixed it, as per a lxd-users mailing list post, is this:

$ lxc network edit lxdbr0

This will edit your configuration in a terminal editor. It will look something like this:

config:
  ipv4.address: 10.216.134.1/24
  ipv4.nat: "true"
  ipv6.address: none
  ipv6.nat: "true"
name: lxdbr0
type: bridge

Add three lines to it:

config:
  ipv4.address: 10.216.134.1/24
  ipv4.nat: "true"
  ipv6.address: none
  ipv6.nat: "true"
  raw.dnsmasq: |
    auth-zone=lxd
    dns-loop-detect
name: lxdbr0
type: bridge

and this should cause dnsmasq, which is being run by lxd, to detect DNS loops. This, at least for me, solved the problem and stopped systemd-resolved and dnsmasq using 100% CPU.

Answer 8

In my case (needing to provide a DNS service to other machines) I was able to solve the problem by telling dnsmasq to only bind to the ethernet interface (systemd-resolvd binds to loop-back) by setting....

...
interface=eth0
...
bind-interfaces

in dnsmasq.conf

Answer 9

I solved it this way:

Add or uncomment the following line in /etc/default/dnsmasq:

IGNORE_RESOLVCONF=yes

Create your own resolv file (/etc/resolv.personal) to define nameservers. You can use any nameserver here. I took two from https://www.opennic.org

nameserver 5.132.191.104
nameserver 103.236.162.119

In /etc/dnsmasq.conf add or uncomment the following line:

resolv-file=/etc/resolv.personal

Then restart dnsmasq and disable the default resolver: systemd-resolved.

sudo service dnsmasq restart

sudo systemctl stop systemd-resolved
sudo systemctl disable systemd-resolved

Answer 10

I'm not sure why both services are trying to use the same address. Maybe you can arrange them as in my case on Xubuntu 18.04.1, where their configuration is the following:

xy@zq:~$ sudo netstat -tulpn | grep 53
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      13549/systemd-resol 
tcp        0      0 127.0.0.1:53            0.0.0.0:*               LISTEN      9632/dnsmasq 

systemd-resolved works like this by default (it only listens on 127.0.0.53), but dnsmasq tries to bind the wildcard address by default. To allow them to work side-by-side as seen above, set the following dnsmasq options (either in /etc/dnsmasq.conf or /etc/dnsmasq.d/pick-your-own-filename.conf):

listen-address=127.0.0.1
bind-interfaces

To make systemd-resolved using my dnsmasq I just set:

#/etc/systemd/resolved.conf 
[Resolve]
DNS=127.0.0.1

In my dnsmasq config I set my external nameservers:

#/etc/dnsmasq.conf
nameserver x.x.x.x
nameserver y.y.y.y

After restarting everything:

# sudo systemctl restart systemd-resolved.service
# sudo systemctl restart dnsmasq.service

systemd-resolved will set the default DNS server to dnsmasq in:

#/etc/resolv.conf
nameserver 127.0.0.1

Answer 11

I was unable to get dnsmasq to start using the solutions found online, i.e. disabling systemd-resolved, changing dnsmasq.conf to do "bind dynamic" instead of "bind interfaces". I was able to get it to start at boot by having dnsmasq start After network-online.service rather than network.service:

[Unit]
Description=dnsmasq - A lightweight DHCP and caching DNS server
Requires=network.target
Wants=nss-lookup.target
Before=nss-lookup.target
After=network-online.target #This line changed

Answer 12

Here's what worked for me (after hours of pain) in Ubuntu 18.10 Cosmic Cuttlefish. I did this to take advantage dnsmasq's comparatively more robust caching mechanism and to avoid NGINX resolver vulnerabilities. Note that I'm using the Ubuntu Server edition (no NetworkManager/nmcli, just systemd-networkd) and this is running on AWS EC2, so I also needed to keep DNS and DHCP working with the default EC2 search domain. I didn't want to disable systemd-resolved entirely because I have no idea how that might affect future updates. Everything here is run as root/sudo unless otherwise noted (this happens by default when passed in as EC2 User Data).

## Configure dnsmasq to work with systemd-resolved
# Set static hostname with hostnamectl
hostnamectl set-hostname mydomainname
# Add an entry for the hostname to /etc/hosts
tee --append /etc/hosts <<EOF
127.0.0.1 mydomainname
EOF
# Disable stub listener for resolvconf and set DNS to loopback
tee --append /etc/systemd/resolved.conf <<EOF
DNSStubListener=no
DNS=127.0.0.1
EOF
# Tell dnsmasq to ignore resolvconf
tee --append /etc/default/dnsmasq <<EOF
IGNORE_RESOLVCONF=yes
EOF
# Create dropin directory
mkdir -p /etc/systemd/system/dnsmasq.service.d
# Create systemd dropin to make sure systemd-resolved stops before dnsmasq starts
tee /etc/systemd/system/dnsmasq.service.d/resolved-fix.conf <<EOF
[Unit]
After=systemd-resolved.service
[Service]
ExecStartPre=bin/systemctl stop systemd-resolved.service
ExecStartPost=bin/systemctl start systemd-resolved.service
EOF
# Create custom resolvconf with name servers (I usec cloudflare)
tee /etc/resolv.mydomainname <<EOF
nameserver 1.1.1.1
nameserver 1.0.0.1 
nameserver [2606:4700:4700::1111] 
nameserver [2606:4700:4700::1001] 
EOF
# Configure dnsmasq
tee /etc/dnsmasq.d/mydomainname.conf <<EOF
# Region comes from:
# EC2_AVAIL_ZONE=$(curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone)
# EC2_REGION=${EC2_AVAIL_ZONE%?}
domain=$EC2_REGION.compute.internal
resolv-file=/etc/resolv.mydomainname
listen-address=127.0.0.1
port=53
interface=lo
bind-dynamic
domain-needed
bogus-priv
dnssec
dns-forward-max=300
cache-size=1000
neg-ttl=3600
EOF
# Reload to pick up dropin
systemctl daemon-reload
# Stop systemd-resolved
systemctl stop systemd-resolved
# Start dnsmasq
systemctl restart dnsmasq

Verify that 127.0.0.1#53 is being used for resolution and DNSSEC is working with something like dig +trace facebook.com