Is it possible to force fwmark reflection in arbitrary-TCP reply packets?

Question

Has anyone solved the problem of fwmarks being lost in TCP socket-connection replies?

Background:

I need to ensure replies go out on the same interface on which their requests were received, and need to support the networksoutbound interface not being distinguishable by any other means, such as source-based routing; so this answer started me looking at policy-based routing. At first, did not seem to work for my case at all (used tcpdump to watch the traffic on the interface as well as directing iptables to log packet metadata going both directions), and at that point, I saw that the incoming packets were indeed being marked according to the incoming interface, as I desired. But the outgoing packets did not retain the same mark! I reduced my test case from a socket connection to a simple ping (ICMP echo), and when I realized the fwmark was being lost, I did more research and discovered the "fwmark_reflect" sysctl parameter. Enabled that, and it did resolve the issue for my simplified test case, but of course, since that sysctl only relates to "kernel-originated IP packets that have no user socket associated with them", this does not help with the original problem: establishing socket connections, such as ssh...

More backing detail:

iptables packet-marking rules:

-A PREROUTING -i ens3 -j MARK --set-mark 1 -A PREROUTING -i ens4 -j MARK --set-mark 2

Routing rules (some are duplicates because I was trying anything, out of desperation): #ip rule list 0: from all lookup local 0: from all fwmark 0x1 lookup mgmt 0: from all fwmark 0x2 lookup cust 10: from all fwmark 0x1 lookup mgmt 20: from all fwmark 0x2 lookup cust 32766: from all lookup main 32767: from all lookup default 65536: from all fwmark 0x2 oif ens4 lookup cust

#ip route list table mgmt
  default via 10.100.16.1 dev ens3 
#ip route list table cust

default via 10.100.16.1 dev ens4 ip route get 221.194.47.243 mark 2 221.194.47.243 via 10.100.16.1 dev ens4 src 10.100.16.13 mark 2

Iptables' LOG output for echo-request-and reply(full round-trip is complete; remote does reflect successful receipt):

[27314.322482] pIng IN=ens4 OUT= MAC=00:00:17:01:8c:ff:00:00:17:01:cf:b5:08:00 SRC=10.100.21.143 DST=10.100.16.13 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=18269 DF PROTO=ICMP TYPE=8 CODE=0 ID=14889 SEQ=1 MARK=0x2

[27314.323446] pOng IN= OUT=ens4 SRC=10.100.16.13 DST=10.100.21.143 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=39688 PROTO=ICMP TYPE=0 CODE=0 ID=14889 SEQ=1 MARK=0x2

Similar logged metadata while attempting a socket connection to port 22(remote reports connection failure, because of course with the original fwmark lost, the routing table tries to send it out the wrong interface):

[23305.165235] ens4-PRERT IN=ens4 OUT= MAC=00:00:17:01:8c:ff:00:00:17:23:a9:ae:08:00 SRC=156.151.8.14 DST=10.100.16.13 LEN=60 TOS=0x10 PREC=0x20 TTL=41 ID=45069 DF PROTO=TCP SPT=5874 DPT=22 WINDOW=14600 RES=0x00 SYN URGP=0 MARK=0x2

[23305.169993] output IN= OUT=ens3 SRC=10.100.16.13 DST=156.151.8.14 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=22 DPT=5874 WINDOW=26844 RES=0x00 ACK SYN URGP=0

Apologies for the length of this post. I have been beating my head against the wall of this problem for quite a while, and could not come up with think of a more terse way to communicate all the relevant detail. Would greatly appreciate any guidance anyone can give, even if it is "this is not possible". Cheers!

Answer 1

Following on my comment, here is I think a solution that should work as you intend. Please note that the order is the reverse of the examples in the Netfilter Connmark link, but since your decisions are made with incoming packets, not outgoing, that's the right order to go. So remove your two previous MARK rules (I put them back here) and replace with:

# iptables -t mangle -A INPUT -i ens3 -j MARK --set-mark 1
# iptables -t mangle -A INPUT -i ens4 -j MARK --set-mark 2    
# iptables -t mangle -A INPUT -j CONNMARK --save-mark

# iptables -t mangle -A OUTPUT -j CONNMARK --restore-mark

I'm using INPUT (which is called after PREROUTING, but only for local flows) for the sake of symmetry and clarity: this is for local connections, not forwarding. Using PREROUTING instead of INPUT (but keeping OUTPUT) would give the same results.

This uses conntrack, which is tracking all flows, to remember the mark, and put it back in the locally generated outgoing packet, because conntrack knows it's part of the same flow.

Using an OUTPUT rule in the mangle table triggers a reroute check (see this Linux routing and netfilter schematic), and that's what is needed to route according to the mark.

If you're using other -t mangle rules or if the server is also forwarding, the rules will have to be changed. Anyway the --save-mark should probably be last in INPUT, and the --restore-mark be first in OUTPUT.

The mark is even set for non-user reply packets (like TCP RST), so fwmark_reflect might not even be needed anymore or might interfere, you should test.