LXC, Docker and iptables on the same host, timeout on port-forwarding from the container to the container

Question

thanks for reading! I have a publicly exposed server (debian 10), used as an host, running both LXC (not LXD) and Docker. These two container services are running with a mostly original configuration (LXC using lxc-net and its lxcbr0 bridge, Docker using its docker0 bridge), all containers are NAT-ed.

Almost everything is OK (internet access in container, port-forward, etc.), excepted that I can't self-access a container from inside, using the host public IP (and so the port-forward) when docker is started (with iptables: true).

Example: I want to access a website hosted in the LXC container web, with its IP pointing to the physical host.
From outside (eg. my computer): it works
From the host itself: it works
From another LXC container: it works
From a Docker container: it works
From the web container itself: timeout

The LXC containers are NET-ed using simple home made iptables rules.

iptables -t nat -A PREROUTING -d ${MY_PUBLIC_IP}/32 -p tcp -m tcp --dport 443 -j DNAT --to-destination 10.0.0.10:443
iptables -t nat -A OUTPUT -o lo -p tcp -m tcp --dport 443 -j DNAT --to-destination 10.0.0.10:443
iptables -t nat -A POSTROUTING -s 10.0.0.0/8 ! -d 10.0.0.0/8 -j MASQUERADE # added by LXC

It's been a few weeks I'm desperately trying to fix out this problem without success. I tried to tweak these rules, to add a few MASQUERADE rules, to LOG the rules, but there is probably a misconception from myself that I'm unable to see.
Running docker with iptables: false or stopping docker fixes the issue.
One workaround could be to tweak my /etc/hosts to use the loopback interface from the container, but I'd prefer to avoid, since I have too much different domains.

My guess, maybe, is that maybe the DOCKER-USER rule is blocking my request but I'm unable to find a workaround.

Any indications on what I am doing wrong from someone more experienced than me?

Answer 1

You need to have an hairpinning configuration for this to work properly.

That it's working for other LXC containers in the same LAN when it should probably not is also explained below, as well as all the wonderful weird world of bridge netfilter.

---

You are subject to two different effects, one very non-obvious and altering the way of understanding all this.

• Docker causes the kernel module br_netfilter to be loaded. As I'm explaining in an answer to How does iptable work with linux bridge?, this causes iptables rules to affect bridged frames.

  Normally this rule should be called from the routing path:

  iptables -t nat -A PREROUTING -d ${MY_PUBLIC_IP}/32 -p tcp -m tcp --dport 443 -j DNAT --to-destination 10.0.0.10:443
  

  and will turn the destination into 10.0.0.10. This also means that:

  iptables -t nat -A POSTROUTING -s 10.0.0.0/8 ! -d 10.0.0.0/8 -j MASQUERADE # added by LXC
  

  will now not trigger.

  So if the client container is 10.0.0.11 this would result in the web container answering directly to 10.0.0.11 by bridge path. 10.0.0.11 would receive a connection from an unknown IP address 10.0.0.10 (and not MY_PUBLIC_IP) so would send back a TCP RST. That means that without Docker having been run (even if not involved at all in the connection), the LXC -> LXC case wouldn't have worked either.

  With br_netfilter loaded the bridged frame is also subject to conntrack and NAT. So 10.0.0.10 when replying to 10.0.0.11 is intercepted in the bridge path and de-DNATed back to source MY_PUBLIC_IP. 10.0.0.11 sees the reply still comming from MY_PUBLIC_IP: transparent hairpinning happened.

  While this is nice, this is a weird behaviour you have to keep in mind and not take for granted. The normal behaviour is to have bridged packets not be changed. Likewise the normal behaviour is to have iptables' FORWARD chain not filter the bridged containers, but they are now filtered.

  For more informations on this, check this schematic. The green (network level) boxes in the blue (bridge level) background at the bottom are enabled by br_netfilter. You can also check figure 3c in ebtables/iptables interaction on a Linux-based bridge.

• For a connection from the web container to itself with br_netfilter at some time (even without the module loaded) there would be a connection from 10.0.0.10 to 10.0.0.10 over the wire which can't work: by default the network stack drops remote packets received with its own source address. But there is an other issue that looks like a bug. See below.

---

So to apply a band-aid to the current configuration, you must change the source IP address to any routed IP address. It could be anything, even 8.8.8.8, but better keep one controlled by oneself: either MY_PUBLIC_IP or if not practical, the address assigned on lxcbr0 (which I will assume here to be 10.0.0.1). That means the web service will not see its own IP address in its logs. That's quite unavoidable.

But, for reason I can't explain and after trial and error (tracing packets in netfilter), one also have to enable promiscuous mode on the bridge's self interface lxcbr0, else the packet disappears between ip nat/PREROUTING and bridge filter/INPUT (in this order, check again figure 3c) and thus never reaches ip nat/POSTROUTING. It also seems that the behaviour described in chapter 8 doesn't happen as expected.

ip link set lxcbr0 promisc on

and then just add:

iptables -t nat -A POSTROUTING -s 10.0.0.10 -p tcp --dport 443 -j SNAT --to-source $MY_PUBLIC_IP

or:

iptables -t nat -A POSTROUTING -s 10.0.0.10 -p tcp --dport 443 -j MASQUERADE

or else, just by matching a DNATed flow (it could also use the SNAT target):

iptables -t nat -A POSTROUTING -s 10.0.0.10 -m conntrack --ctstate DNAT -j MASQUERADE

---

Newer kernels (I believe >= 5.3) allow to selectively enable or disable, per network namespace or even per-bridge (eg: ip link set lxcbr0 type bridge nf_call_iptables 1), the activation of net.bridge.bridge-nf-call-iptables. But without knowing how Docker triggers this (usually because of a -m physdev iptables match, but maybe not the only case), it's difficult to know where to disable this feature while keeping Docker working as usual.