Route traffic in a cgroup outside a VPN tunnel

Question

I'm trying to get some traffic to go through a VPN and other traffic to not do so.

Packets with a given fwmark are supposed to go to my default interface (wlo1), and all other traffic to a tunnel interface (tun0, using OpenVPN) in the main table. I have added this rule in nftables:

table ip test {
    chain test {
        type route hook output priority mangle; policy accept;
        meta cgroup 1234 meta mark set 1
    }
}

In the main routing table I have these entries:

default via 10.11.0.1 dev tun0 
10.11.0.0/16 dev tun0 proto kernel scope link src 10.11.0.20 
[WANIP.0]/24 dev wlo1 proto kernel scope link src [WANIP] 
128.0.0.0/1 via 10.11.0.1 dev tun0 
[VPNIP] via [WANGATEWAY] dev wlo1

Packets with fwmark 1 is instead led to their own routing table: ip rule add from all fwmark 1 lookup test. In the test table I have added the following route:

default via [WANGATEWAY] dev wlo1

When I run ping 8.8.8.8 from this cgroup, it's stuck. It seems able to send but not receive any packets.

VPN traffic works as expected.

What exactly is going on?

Answer 1

When a packet is emitted, there's a routing decision made: this decision chooses the outgoing interface and the matching source IP address to use.

When the route/output chain sets a mark, it triggers a reroute check, as seen in this schematic (which was made for iptables in mind but is totally usable for nftables). The reroute check alters the route... but doesn't change the source IP address. So more work has to be done.

• add a NAT rule to alter the source IP address.

This has to be done after the reroute check, so it's done in nat/postrouting. Note that the same table can have different chain types (contrary to iptables where table <=> type).

    nft add chain ip test testnat '{ type nat hook postrouting priority srcnat; policy accept; }'
    nft add rule ip test testnat meta mark 1 masquerade

Now the correct packet really leaves.

• and allow reply flow to be accepted despite not arriving through the default route

1. You can either relax Reverse Path Forwarding to Loose mode by changing rp_filter:

       sysctl -w net.ipv4.conf.wlo1.rp_filter=2
   

2. or mark reply flow to use the same routing table as outgoing flow. It won't really improve security but anyway:

       nft add chain ip test testpre '{ type filter hook prerouting priority mangle; policy accept; }'
       nft add rule ip test testpre iif "wlo1" meta mark set 1
   

Alas, this won't work without yet-an-other-tweak which is needed since a new undocumented feature appeared in 2010:

        sysctl -w net.ipv4.conf.wlo1.src_valid_mark=1

Notes:

• It's possible to do better security-wise by storing the mark in conntrack's connmark (ct mark) to only allow the correct reply flows and not anything else to bypass the Strict Reverse Path Forwarding. In Strict mode, nothing will reach through wlo1 anymore unless it's a reply from outgoing traffic. Here's the full corresponding nftables rule file for this (to be used with option 2. above while replacing nftables rules):

    table ip test {
            chain test {
                    type route hook output priority mangle; policy accept;
                    meta cgroup 1234 meta mark set 1 ct mark set meta mark
            }
  
            chain testnat {
                    type nat hook postrouting priority srcnat; policy accept;
                    meta mark 1 masquerade
            }
  
            chain testpre {
                    type filter hook prerouting priority mangle; policy accept;
                    ct mark 1 meta mark set ct mark
            }
    }
  

• Also, the mark reroute check might interfere with correct PMTU / TCP MSS in some cases, according to what I could read there in the uidrange entry: https://kernelnewbies.org/Linux_4.10#Networking

• If you can convert the cgroups usage into a limited set of uids instead, then this can be done correctly using only the routing stack, without netfilter or nftables, using ip rule add ... uidrange ... as described above. See my answer there about it: Routing traffic for a user through specific interface (tum1).