I have a L4 ipvs load balancer with L7 envoy balancers setup. Let's say one of my L4 balancers goes down and thanks to consistent hashing the traffic which is now handled (thanks to BGP) by another L4 balancer is proxied to the same L7 node.

This should work without any problems and I would think is a common setup.

Problem is with long-running connections. When new L4 node receives the traffic (just data - ACK/PUSH packets) and no SYN packet has been received by the node, the node just sends RST packet to the client which terminates the connection. Picture below illustrates this.

This should not be happening and my question is, is there a way (a sysctl config or something) which is the reason for this? I know I can perhaps drop RST packets using iptables, but that doesn't sound right.

enter image description here

  • If the node sent a RST that means it doesn't recognize the connection as an already established connection. You should compare network captures before and after the fail over to try and see what difference can be seen. It could be a local firewall that doesn't receive updated conntrack entries from peers that rejected it (it's possible to synchronize flows with the conntrackd daemon). Anyway I'm not sure at all of how is working your setup with your description. – A.B Mar 11 '20 at 12:16
  • Oh sorry, I wasn't sure which site should be used. I've just deleted it from SO. – Diavel Mar 13 '20 at 5:49
  • I understand why it's sending the resets, but can't figure out how to fix that. The conntrack is somehow integrated into the ipvs module so there is no easy way to disable that. Also it's not synchronized between the L3/L4 nodes. Do you think that could be the source of the problem? The thing is, I have deleted all iptables rules and haven't found a sysctl setting related to that :/ – Diavel Mar 13 '20 at 6:02

There is actually a sysctl variable net.ipv4.vs.sloppy_tcp for this specific problem (https://lore.kernel.org/patchwork/patch/386081/?fbclid=IwAR17t0jEvRSlvZFch1Lz_CDMjYOzUluuNGQmiyKequZK1Vq4kI75vezWEGs) which solves this.

The big thank you goes to Patric Shuff who's helped me figure this out (great presentaition on this topic - https://www.usenix.org/conference/lisa16/conference-program/presentation/shuff).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.