0

We have small kuberentes cluster running (CentOS 7, Kuberenetes 1.13 + Flannel) and after some tweaking TCP configuration (see below), we noticed that DNS was not working properly.

I don't think that our changes are directly responsible for what I have observed, nor that kubernetes is responsible. I looked up in IP tables and AFAIK everything looked good. What I observed was following:

  1. Pod 10.23.118.10 send UDP(53) package to DNS ClusterIP 10.22.0.10
  2. Destination IP of package is changed then from ClusterIP (10.22.0.10) to the IP of DNS's server pod (10.23.118.2) (DNAT)
  3. Server gets request, process it and then send response back to 10.23.118.10
  4. At this point netfilter should replace source IP 10.23.118.2 with 10.22.0.10 before it forwards package, but for some reason it does not do it
  5. Libc receives package and rejects it because it sees that response came from 10.23.118.2 instead 10.22.0.10 or we get ICMP package, saying to port is unreachable.

enter image description here

What is strange about this, is that it only happens if DNS request is sent to pod that is running on same machine. If DNS request came from pod running on other machine, everything worked fine.

I suppose that we are not the only one seeing this. Did you had similar situation? I am not sure whenever this is a bug in linux's netfilter or docker/kubernetes breaks something when configuring bridge interfaces. Where should I look for more information?

Here is TCP configuration we tried to apply:

net.core.somaxconn = 1000
net.core.netdev_max_backlog = 5000
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_wmem = 4096 12582912 16777216
net.ipv4.tcp_rmem = 4096 12582912 16777216
net.ipv4.tcp_max_syn_backlog = 8096
net.ipv4.tcp_slow_start_after_idle = 0
0

The issue seems to be CentOS 7 default configuration.

When reloading configuration sysctl -p --system, it will also reload /usr/lib/sysctl.d/00-system.conf. In /usr/lib/sysctl.d/00-system.conf, we can see following:

# Disable netfilter on bridges.
net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-arptables = 0

Once this is done (netfilter disabled on bridge), NAT-ing will not work properly on docker0 bridge.

Interestingly, once you restart docker and query system with sysctl -a, you can see following:

net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1

I can only assume that this settings are changed by docker itself during start.

Which explains why restarting docker usually helps.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.