2

I've just spent about an hour trying to understand why my docker containers couldn't connect to the internet, in particular couldn't resolve any DNS names. By now I've found a workaround, but I still don't understand this.

The symptom was that according to Wireshark, packets would be sent out over the virtual docker network, but never get transmitted over my physical NIC, despite the fact that I had enabled IPv4 forwarding. According to iptables packet counters, they were recorded in the MASQUERADE rule of the POSTROUTING chain of the nat table, but not in the FORWARD chain of the filter table.

I've got the following network devices involved here:

  • net0 is my physical LAN device, with IP 192.168.1.5, located behind a DSL router at 192.168.1.1
  • tun1 is a VPN link to my office network
  • docker0 is a virtual bridge device which docker set up automatically, with docker containers using the 172.17.0.0/16 network and the bridge at 172.17.42.1

The key point is the VPN. When I'm at my office, I want to be able to connect to my computer through a DynDNS record set up by my DSL router. So The packets from my office to my computer would not go through the VPN but instead through the NAT in the router. Which means that in order to establish a connection, the reply packets must follow the reverse path, not going through the VPN either. To achieve this, I've set up a special routing table for outgoing packets coming from my LAN:

# ip rule list
0:      from all lookup local 
50:     from all to 127.0.0.0/8 lookup main 
51:     from all to 192.168.1.0/24 lookup main 
100:    from 192.168.1.0/24 lookup net0 
32766:  from all lookup main 
32767:  from all lookup default 

# ip route list table net0
default via 192.168.1.1 dev net0 
192.168.1.0/24 dev net0  scope link 

# ip route list table main
default via 192.168.1.1 dev net0  metric 3 
10.0.0.0/8 dev tun1  scope link 
127.0.0.0/8 dev lo  scope host 
127.0.0.0/8 via 127.0.0.1 dev lo 
<VPN gateway> via 192.168.1.1 dev net0  src 192.168.1.5 
<VPN network> dev tun1  scope link 
172.17.0.0/16 dev docker0  proto kernel  scope link  src 172.17.42.1 
192.168.1.0/24 dev net0  proto kernel  scope link  src 192.168.1.5  metric 3 

Eventually I found out that adding another rule to use the main table for packets to the docker network was sufficient to make this work. But why? I can see how the special routing table might have affected reply packets coming in from my router. But why did it affect outgoing packets, causing them to vanish completely? And is there a more flexible way to express the fact that I want to route every reply packet the same way the response packet came in, so that I don't have to add rules whenever I set up a new virtual network of some kind?

0

Don't you just want to fix the root cause of the problem by making sure you don't sidestep the VPN link? If so, why not create a separate set of DNS entries that you can use while the VPN is connected? You can put 192.168. and 172.17. in A records in public DNS, so they're still resolvable through your office DNS servers.

|improve this answer|||||
  • On my laptop I have a ssh config which to me essentially means “get me a link home, wherever I am now”. I wouldn't want to use different commands depending on whether the office side of the VPN might be available or not. Same goes for some other protocols which I expect to work without reconfiguration both from my office and while I'm on the road. So no, I don't want to fix this aspect. Your answer may well be valuable to others in similar situations, though. – MvG Jun 22 '15 at 21:17
  • @MvG You can actually make it work like that. You can put all the VPN DNS names in a subdomain, so for example you have foo.vpn.example.com pointing to the internal IP and foo.example.com pointing to the external IP. Make your ssh config invoke a small shell wrapper that checks for the existence of a VPN interface at work, in which case it connects to %h.vpn.example.com, otherwise it connects to %h.example.com. – Josip Rodin Jun 23 '15 at 8:58
  • As for other protocols, some clients may be sufficiently resilient if you have a round-robin DNS entry. For example, make imap.example.com serve both the external and the internal IP, and whenever one connection attempt fails due to connection refused, the client will immediately proceed to the next IP and succeed. If anything, you can easily experiment if it works for you. – Josip Rodin Jun 23 '15 at 9:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.