1

I have a VXLAN tunnel between two Linux hosts. This tunnel is carried over an Ethernet link with an MTU of 1500 bytes. At each end, the VXLAN interface is slaved to a bridge, along with another Ethernet interface. It looks something like this:

                 |         Host A               |                 Host B       |
Client Device  <--> eth0 <--> br0 <--> vxlan0 <-|-> vxlan0 <--> br0 <--> eth0 <--> Client Device
                 |                              |                              |
                 |                       eth1 <-|-> eth1                       |

Here only the client devices and eth1 have IP configuration - everything else is just a layer 2 link between two sites.

VXLAN involves 50 bytes of overhead. To avoid breaking certain protocols, I've set the MTU on the layer 2 link (eth0, br0, vxlan0) to be 1500 bytes. To avoid TCP fragmentation, I want to set the TCP MSS on the vxlan0 interfaces to be 1400 bytes. How can I do this?

I know that I can turn iptables on for the bridges (echo 1 > /sys/class/net/br0/bridge/nf_call_iptables). However, iptables has a lot of other rules that are unrelated to this part of the system and doing this comes with a considerable performance cost.

I know that I can set the MSS for a route with ip route add ... advmss 1400. But note that none of the packets involved are routed on Host A or Host B. They are just bridged through from interface to another.

Is there some way to apply a lower TCP MSS to this traffic without incurring the iptables performance penalty?

3
  • Wouldn't you need to set MSS at the client devices, where the TCP packets ate segmented?
    – haggai_e
    Oct 20 '20 at 0:06
  • 1
    @haggai_e: No. When the TCP stream starts, one client device sends a TCP SYN packet to the other client with the default MSS set. Any part of the link between them can reduce the MSS as the SYN packet traverses the link. The other client will then reply with an ACK containing the minimum MSS across the whole link. This is how TCP streams find out what segment size to send to avoid fragmentation, even when the local MTU could support a larger segment size.
    – Tom
    Oct 20 '20 at 9:01
  • Makes sense, thanks @Tom.
    – haggai_e
    Oct 20 '20 at 15:11
1

nftables aims to replace ebtables + iptables: most features are also available at the bridge level the same way they are available at the IP level without using br_netfilter (including stateful IP firewalling at bridge level, with kernel >= 5.3).

Here's a ruleset (let's call it bridgemss.nft) to load on HostA and HostB with nft -f bridgemss.nft:

table bridge t
delete table bridge t

table bridge t {
    chain c {
        type filter hook forward priority filter; policy accept;
        oifname vxlan0 tcp flags syn tcp option maxseg size set 1400
    }
}

Of course the ruleset can be changed to suit the needs (eg: have it used only on Host A with both directions instead of having it on Host A and Host B in a single direction each).

Should HostA and HostB assign an address on br0 to communicate over VXLAN instead of directly with eth1, then this rule should be duplicated for example in bridge output hooks (or on one system output + input hooks) since this won't count as forwarded frames.

3
  • Thanks, that's very useful to know. We're currently stuck with a 4.X kernel but this only adds to the impetus to bring it up to date.
    – Tom
    Oct 20 '20 at 9:02
  • AFAIK, the ruleset above "just" requires kernel >= 4.10 to handle properly the tcp checksums.
    – A.B
    Oct 20 '20 at 10:28
  • Also useful to know, thanks. Sadly, "4.X" actually means "4.9".
    – Tom
    Oct 20 '20 at 10:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.