I've just spent about an hour trying to understand why my docker containers couldn't connect to the internet, in particular couldn't resolve any DNS names. By now I've found a workaround, but I still don't understand this.
The symptom was that according to Wireshark, packets would be sent out over the virtual docker network, but never get transmitted over my physical NIC, despite the fact that I had enabled IPv4 forwarding. According to
iptables packet counters, they were recorded in the
MASQUERADE rule of the
POSTROUTING chain of the
nat table, but not in the
FORWARD chain of the
I've got the following network devices involved here:
- net0 is my physical LAN device, with IP 192.168.1.5, located behind a DSL router at 192.168.1.1
- tun1 is a VPN link to my office network
- docker0 is a virtual bridge device which docker set up automatically, with docker containers using the 172.17.0.0/16 network and the bridge at 172.17.42.1
The key point is the VPN. When I'm at my office, I want to be able to connect to my computer through a DynDNS record set up by my DSL router. So The packets from my office to my computer would not go through the VPN but instead through the NAT in the router. Which means that in order to establish a connection, the reply packets must follow the reverse path, not going through the VPN either. To achieve this, I've set up a special routing table for outgoing packets coming from my LAN:
# ip rule list 0: from all lookup local 50: from all to 127.0.0.0/8 lookup main 51: from all to 192.168.1.0/24 lookup main 100: from 192.168.1.0/24 lookup net0 32766: from all lookup main 32767: from all lookup default # ip route list table net0 default via 192.168.1.1 dev net0 192.168.1.0/24 dev net0 scope link # ip route list table main default via 192.168.1.1 dev net0 metric 3 10.0.0.0/8 dev tun1 scope link 127.0.0.0/8 dev lo scope host 127.0.0.0/8 via 127.0.0.1 dev lo <VPN gateway> via 192.168.1.1 dev net0 src 192.168.1.5 <VPN network> dev tun1 scope link 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.42.1 192.168.1.0/24 dev net0 proto kernel scope link src 192.168.1.5 metric 3
Eventually I found out that adding another rule to use the
main table for packets to the docker network was sufficient to make this work. But why? I can see how the special routing table might have affected reply packets coming in from my router. But why did it affect outgoing packets, causing them to vanish completely? And is there a more flexible way to express the fact that I want to route every reply packet the same way the response packet came in, so that I don't have to add rules whenever I set up a new virtual network of some kind?