You need to have an hairpinning configuration for this to work properly.
That it's working for other LXC containers in the same LAN when it should probably not is also explained below, as well as all the wonderful weird world of bridge netfilter.
You are subject to two different effects, one very non-obvious and altering the way of understanding all this.
Docker causes the kernel module br_netfilter
to be loaded. As I'm explaining in an answer to How does iptable work with linux bridge?, this causes iptables rules to affect bridged frames.
Normally this rule should be called from the routing path:
iptables -t nat -A PREROUTING -d ${MY_PUBLIC_IP}/32 -p tcp -m tcp --dport 443 -j DNAT --to-destination 10.0.0.10:443
and will turn the destination into 10.0.0.10. This also means that:
iptables -t nat -A POSTROUTING -s 10.0.0.0/8 ! -d 10.0.0.0/8 -j MASQUERADE # added by LXC
will now not trigger.
So if the client container is 10.0.0.11 this would result in the web container answering directly to 10.0.0.11 by bridge path. 10.0.0.11 would receive a connection from an unknown IP address 10.0.0.10 (and not MY_PUBLIC_IP) so would send back a TCP RST. That means that without Docker having been run (even if not involved at all in the connection), the LXC -> LXC case wouldn't have worked either.
With br_netfilter
loaded the bridged frame is also subject to conntrack and NAT. So 10.0.0.10 when replying to 10.0.0.11 is intercepted in the bridge path and de-DNATed back to source MY_PUBLIC_IP. 10.0.0.11 sees the reply still comming from MY_PUBLIC_IP: transparent hairpinning happened.
While this is nice, this is a weird behaviour you have to keep in mind and not take for granted. The normal behaviour is to have bridged packets not be changed. Likewise the normal behaviour is to have iptables' FORWARD chain not filter the bridged containers, but they are now filtered.
For more informations on this, check this schematic. The green (network level) boxes in the blue (bridge level) background at the bottom are enabled by br_netfilter
. You can also check figure 3c in ebtables/iptables interaction on a Linux-based bridge.
For a connection from the web container to itself with br_netfilter
at some time (even without the module loaded) there would be a connection from 10.0.0.10 to 10.0.0.10 over the wire which can't work: by default the network stack drops remote packets received with its own source address. But there is an other issue that looks like a bug. See below.
So to apply a band-aid to the current configuration, you must change the source IP address to any routed IP address. It could be anything, even 8.8.8.8, but better keep one controlled by oneself: either MY_PUBLIC_IP or if not practical, the address assigned on lxcbr0 (which I will assume here to be 10.0.0.1). That means the web service will not see its own IP address in its logs. That's quite unavoidable.
But, for reason I can't explain and after trial and error (tracing packets in netfilter), one also have to enable promiscuous mode on the bridge's self interface lxcbr0, else the packet disappears between ip nat/PREROUTING and bridge filter/INPUT (in this order, check again figure 3c) and thus never reaches ip nat/POSTROUTING. It also seems that the behaviour described in chapter 8 doesn't happen as expected.
ip link set lxcbr0 promisc on
and then just add:
iptables -t nat -A POSTROUTING -s 10.0.0.10 -p tcp --dport 443 -j SNAT --to-source $MY_PUBLIC_IP
or:
iptables -t nat -A POSTROUTING -s 10.0.0.10 -p tcp --dport 443 -j MASQUERADE
or else, just by matching a DNATed flow (it could also use the SNAT target):
iptables -t nat -A POSTROUTING -s 10.0.0.10 -m conntrack --ctstate DNAT -j MASQUERADE
Newer kernels (I believe >= 5.3) allow to selectively enable or disable, per network namespace or even per-bridge (eg: ip link set lxcbr0 type bridge nf_call_iptables 1
), the activation of net.bridge.bridge-nf-call-iptables
. But without knowing how Docker triggers this (usually because of a -m physdev
iptables match, but maybe not the only case), it's difficult to know where to disable this feature while keeping Docker working as usual.