I have a Kubernetes cluster running applications (currently on a set of Vagrant CoreOS VMs on a local server)
I want to be able to debug a particular application locally on my laptop, so I worked on setting up VPN into the cluster: a client/server VPN based on
kylemanna/docker-openvpn, deployed as a regular Pod
I created the cert/key pairs, client certs etc... I can connect to the VPN fine.
Now, connecting to the VPN server doesn't get me much if I can't access the Services.
I have the DNS addon running skyDNS in the cluster. I can
nslookup my services from other pods in the cluster, so all that works fine, but I can't resolve Services by name on the VPN client.
I can ping Pods by IP from the VPN client (in the subnet
10.2.0.0/16) but I can't resolve with DNS
nslookup from the client returns:
nslookup myservice 10.3.0.10 Server: 10.3.0.10 Address: 10.3.0.10#53 ** server can't find myservice: SERVFAIL
One of the problems of troubleshooting is that neither
traceroute work on the DNS IP (from any pod), yet it resolves services, so
nslookup is the way I know to check, but that is not very informative.
The VPN host IP the Pod binds to is
The Kubernetes subnet is
The SkyDNS server is at
The VPN server subnet is
On the VPN server ifconfig gives:
eth0 Link encap:Ethernet HWaddr 02:42:0A:02:16:45 inet addr:10.2.22.69 Bcast:0.0.0.0 Mask:255.255.255.0 tun0 Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00 inet addr:10.8.0.1 P-t-P:10.8.0.2 Mask:255.255.255.255
10.2.22.69 is the Pod IP and the VPN Server IP is
10.8.0.1 with the Gateway being
10.8.0.2 i guess.
On the VPN server pod the routign table looks like:
Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface default 10.2.22.1 0.0.0.0 UG 0 0 0 eth0 10.2.22.0 * 255.255.255.0 U 0 0 0 eth0 10.8.0.0 10.8.0.2 255.255.255.0 UG 0 0 0 tun0 10.8.0.2 * 255.255.255.255 UH 0 0 0 tun0 192.168.254.0 10.8.0.2 255.255.255.0 UG 0 0 0 tun0
I can reach my applications by IP (and get data) but couldn't when using the service IP (the proxy IP which is on the 10.3.0.0 subnet)
I added the route
route add -net 10.3.0.0/16 gw 10.8.0.2 to the VPN Server and I can then use the Service IP to get data but the
nslookup just times out then.
I guess the traffic may not be coming back from the DNS. DNS is itself a proxied service in Kubernetes, so that adds a level of complexity. Not sure how to fix this.