2

I'm running a systemd-nspawn container (Debian 9, host also Debian 9) with VirtualEthernet=yes and some ports forwarded, e.g. Port=tcp:443:443. I.e. a web server runs in the container and should be accessible from the outside world.

When I'm activating the ufw firewall, things seem to work fine first, but after some hours, there is no connection from outside possible to the container respectively the web server. When I ufw disable, things work again after a few seconds. I.e. everything is fine, but I don't have a firewall.

3
  • Hello and welcome to the U&L stack exchange site! Please edit your post to include the following information: When you disable ufw, other than not having a firewall do things work as expected with no issue? If so then I believe I have a solution to fix ufw. If your issue shows up again with ufw disabled could you please elaborate as to what service is failing? Is it only port forwarding a service, or is it any type of network connectivity to your containers? Thank you!
    – kemotep
    Commented Sep 11, 2018 at 12:46
  • I added the missing information. Is the problem better explained now?
    – Martin
    Commented Sep 11, 2018 at 20:01
  • I have updated my answer.
    – kemotep
    Commented Sep 12, 2018 at 19:23

2 Answers 2

1

Issue at Hand

You have a systemd-nspawn container running a web server that is being port forwarded. After sometime it no longer has network connectivity, until you disable ufw. I have done some research, hopefully one of these can help fix your issue.

Possible Solutions

I have done more research based on your feedback in the comments. I understand that you say that a bridged interface seems non standard, however every guide I have read about building systemd-nspawn containers involved setting up a bridged virtual interface.

#1 Verify correct networking for Systemd

Following along with this guide, I noticed that they mention to take special care with the networking. Since systemd-nspawn containers are dependent on systemd you need to verify that your host and container use the correct networking.

On your host run the following as root(sudo):

systemctl disable network
systemctl disable networking
systemctl disable NetworkManager
systemctl enable systemd-networkd

Confirm that your server is now using systemd-networkd for networking:

networkctl status lo

● 1: lo
   Link File: /lib/systemd/network/99-default.link
Network File: n/a
        Type: loopback
       State: carrier (unmanaged)
     Address: 127.0.0.1
              ::1

Note that your network should be associated with /lib/systemd/network/99-default.link or thereabouts confirming that you are using systemd-networkd.

From here we create our network interface. Make sure to use the name of your device correctly.

cat > /etc/systemd/network/[nameOfNetworkDevice].network <<EOF
[Match]
Name=[nameOfNetworkDevice]

[Network]
Bridge=br0
IPForward=1
EOF

Create the bridge interface to match the one used earlier, i.e. br0

cat > /etc/systemd/network/br0.netdev <<EOF
[NetDev]
Name=br0
Kind=bridge
EOF

Now create the network for the bridge interface.

cat > /etc/systemd/network/br0.network <<EOF
[Match]
Name=br0

[Network]
DHCP=ipv4
EOF

Now restart networking and move any sysV init configurations to avoid conflicts.

systemctl restart systemd-networkd
mv /etc/network/interfaces /etc/network/interfaces.bak

Start the systemd-resolved service.

systemctl enable systemd-resolved
systemctl start systemd-resolved
ln -sf /run/systemd/resolve/resolv.conf /etc/resolv.conf

Next, test your networking connections and resolving services. Easiest would be to use ping stackexchange.com. Everything should come back nicely now.

Lastly, you want to make sure your containers start on on boot.

systemctl enable machines.target

After this the guide continues to describe building the container which I won't get into.

Now you do not need to do all of this but keep in mind that your issue might be caused in a conflict between how your networking configuration on your host is set up. These steps should allow your host to have network connectivity and bridge it to your containers with no conflict.

#2 Your containers are breaking

I have found several issues ( 1, 2, 3 [) raised on GitHub involving containers intermittently losing network connections.

So please check if these breaks apply to you:

1

Things that are broken under Debian

dnsmasq

/etc/init.d/dnsmasq scripts always injects its own options to dnsmasq command line and will break the configuration

IPTables integration is disabled for systemd (bug #787480)

This means that this nice IPMasquerade=yes option under systemd-networkd and ort=tcp:80:80 port forwarding options do no work and you have to use good old iptables rules like iptables -t nat -A POSTROUTING -s 172.23.0.0/24 -j MASQUERADE to enable NAT and iptables -t nat -A PREROUTING -p tcp -m tcp --dport 80 -m addrtype --dst-type LOCAL -j DNAT --to-destination 172.23.0.2:80 to do port forwarding.

If you are fully up to date on Debian you should avoid some of this, just please check that your dns solution is not being overwritten.

2

Make sure that your iptables are not being flushed. This user was using apf which had a policy of clearing out iptables to generate a new one. You are using ufw but this is still a possibility to lookout for. When you lose networking, does the container no longer have iptables rules?

3

This is again related to docker, however systemd-nspawn may have the same limitations. Check the manpage to see if there is anything wrong with how you are initiating port forwarding. Possibly related to why you lose connectivity after some time is that your firwall rules or port forwarding etc has some kind of rate limiting or connection limitations.

#3 Is your host a laptop or VPS?

According to this stack overflow post, the user Maduraiveeran, was using a VPS that for some reason would run an automated process that messed up the iptables. Your host if on a VPS could be experiencing the same issues.

If you use a laptop, depending on how dhcp and other networking devices are configured you could be losing connections or having your devices renamed as you switch between wifi and ethernet or from wireless access point to wireless access point. Possibly, your network could have some kind of power saving plan that causes issues.

Additional considerations

I am including the link to one of the first blogs I found about nspawn containers to show you how to build one from scratch.

However, I have also found a script for iptables for host port forwarding on the same site. That could help if #1 did not help.

Conclusion

At the end of the day, I am not an expert on containers, but I hope I pointed you in the right direction. Start by troubleshooting how your container gets its networking configuration. Then check that your ufw supports this configuration ( port forwarding, the container virtual interface, etc ). After that, I would make sure your container is built correctly ( can it get networking without port forwarding? Does other services work such as ssh? ). Potentially using a pre-built web server solution that is out there already with the feature set and functions you desire could be a better call. Don't knock trying LXC or docker or rancher or sandstorm. I haven't heard much about systemd-nspawn containers before today. It may behoove you to go with a more supported or at least adopted solution for support.

I am including a link to the manpage for nspawn for additional reference.

Please comment if you have any questions or issues with this answer. I highly suggest you read through each link I have provided thoroughly before attempting the commands. I appreciate feedback to correct any misconceptions and to improve my posts. I can update my answer as needed.

Best of Luck!

6
  • Many thanks for your very elaborate and well-thought answer!
    – Martin
    Commented Sep 12, 2018 at 8:59
  • @Martin, let me know if you have any issues with my post and if any of these fixes work please accept the answer by clicking the check mark next to the up/down arrows on my post so other users know that this is a solution to your question. Thank you.
    – kemotep
    Commented Sep 12, 2018 at 12:21
  • OK, point for point: (1.) is not the problem. The config is correct, and if it were not, then it would not work at all. In my case, function seems to stop only after some hours.
    – Martin
    Commented Sep 12, 2018 at 13:13
  • Next point: (2.) is something I need to investigate further. I need to check how to combine systemd-nspawn with virbr0. This seems not to be the "normal" setup.
    – Martin
    Commented Sep 12, 2018 at 13:16
  • (3.) is all set correctly. As for point (1.), things would not work at all from the start, if it were set wrongly. However, I need to check, whether for some reason the settings might change and lead to the error.
    – Martin
    Commented Sep 12, 2018 at 13:18
0

A quick fix for this

Since this is actually a route rule

What we have to do is make ufw allow all forward

Solution one: Allow all forwarding

edit this file /etc/default/ufw and change this from DROP to ACCEPT

DEFAULT_FORWARD_POLICY="DROP"
<--to-->
DEFAULT_FORWARD_POLICY="ACCEPT"

Then ufw would allow the forward

Solution two:

If you only want specific interface to be routed, instead of make ufw allow forwarding them all, you can add accept rules manually

For example my Vnet interface on host machine is ve-tsingkwai, and the main network interface is enp5s0

what i have to do is

$ tsingkwai@mylaptoplol Sat Jul 16 [16|09] at ~ ->
>>> ufw route allow in on enp5s0 out on ve-tsingkwai

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .