Has anyone solved the problem of fwmarks being lost in TCP socket-connection replies?
I need to ensure replies go out on the same interface on which their requests were received, and need to support the networksoutbound interface not being distinguishable by any other means, such as source-based routing; so this answer started me looking at policy-based routing. At first, did not seem to work for my case at all (used tcpdump to watch the traffic on the interface as well as directing iptables to log packet metadata going both directions), and at that point, I saw that the incoming packets were indeed being marked according to the incoming interface, as I desired. But the outgoing packets did not retain the same mark! I reduced my test case from a socket connection to a simple ping (ICMP echo), and when I realized the fwmark was being lost, I did more research and discovered the "fwmark_reflect" sysctl parameter. Enabled that, and it did resolve the issue for my simplified test case, but of course, since that sysctl only relates to "kernel-originated IP packets that have no user socket associated with them", this does not help with the original problem: establishing socket connections, such as ssh...
More backing detail:
iptables packet-marking rules:
-A PREROUTING -i ens3 -j MARK --set-mark 1 -A PREROUTING -i ens4 -j MARK --set-mark 2
Routing rules (some are duplicates because I was trying anything, out of desperation): #ip rule list 0: from all lookup local 0: from all fwmark 0x1 lookup mgmt 0: from all fwmark 0x2 lookup cust 10: from all fwmark 0x1 lookup mgmt 20: from all fwmark 0x2 lookup cust 32766: from all lookup main 32767: from all lookup default 65536: from all fwmark 0x2 oif ens4 lookup cust
#ip route list table mgmt default via 10.100.16.1 dev ens3 #ip route list table cust
default via 10.100.16.1 dev ens4 ip route get 220.127.116.11 mark 2 18.104.22.168 via 10.100.16.1 dev ens4 src 10.100.16.13 mark 2
Iptables' LOG output for echo-request-and reply(full round-trip is complete; remote does reflect successful receipt):
[27314.322482] pIng IN=ens4 OUT= MAC=00:00:17:01:8c:ff:00:00:17:01:cf:b5:08:00 SRC=10.100.21.143 DST=10.100.16.13 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=18269 DF PROTO=ICMP TYPE=8 CODE=0 ID=14889 SEQ=1 MARK=0x2
[27314.323446] pOng IN= OUT=ens4 SRC=10.100.16.13 DST=10.100.21.143 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=39688 PROTO=ICMP TYPE=0 CODE=0 ID=14889 SEQ=1 MARK=0x2
Similar logged metadata while attempting a socket connection to port 22(remote reports connection failure, because of course with the original fwmark lost, the routing table tries to send it out the wrong interface):
[23305.165235] ens4-PRERT IN=ens4 OUT= MAC=00:00:17:01:8c:ff:00:00:17:23:a9:ae:08:00 SRC=22.214.171.124 DST=10.100.16.13 LEN=60 TOS=0x10 PREC=0x20 TTL=41 ID=45069 DF PROTO=TCP SPT=5874 DPT=22 WINDOW=14600 RES=0x00 SYN URGP=0 MARK=0x2
[23305.169993] output IN= OUT=ens3 SRC=10.100.16.13 DST=126.96.36.199 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=22 DPT=5874 WINDOW=26844 RES=0x00 ACK SYN URGP=0
Apologies for the length of this post. I have been beating my head against the wall of this problem for quite a while, and could not come up with think of a more terse way to communicate all the relevant detail. Would greatly appreciate any guidance anyone can give, even if it is "this is not possible". Cheers!