We have small kuberentes cluster running (CentOS 7, Kuberenetes 1.13 + Flannel) and after some tweaking TCP configuration (see below), we noticed that DNS was not working properly.
I don't think that our changes are directly responsible for what I have observed, nor that kubernetes is responsible. I looked up in IP tables and AFAIK everything looked good. What I observed was following:
- Pod 10.23.118.10 send UDP(53) package to DNS ClusterIP 10.22.0.10
- Destination IP of package is changed then from ClusterIP (10.22.0.10) to the IP of DNS's server pod (10.23.118.2) (DNAT)
- Server gets request, process it and then send response back to 10.23.118.10
- At this point netfilter should replace source IP 10.23.118.2 with 10.22.0.10 before it forwards package, but for some reason it does not do it
- Libc receives package and rejects it because it sees that response came from 10.23.118.2 instead 10.22.0.10 or we get ICMP package, saying to port is unreachable.
What is strange about this, is that it only happens if DNS request is sent to pod that is running on same machine. If DNS request came from pod running on other machine, everything worked fine.
I suppose that we are not the only one seeing this. Did you had similar situation? I am not sure whenever this is a bug in linux's netfilter or docker/kubernetes breaks something when configuring bridge interfaces. Where should I look for more information?
Here is TCP configuration we tried to apply:
net.core.somaxconn = 1000 net.core.netdev_max_backlog = 5000 net.core.rmem_max = 16777216 net.core.wmem_max = 16777216 net.ipv4.tcp_wmem = 4096 12582912 16777216 net.ipv4.tcp_rmem = 4096 12582912 16777216 net.ipv4.tcp_max_syn_backlog = 8096 net.ipv4.tcp_slow_start_after_idle = 0