Iptables MASQUARADE seems to return the "RESULT" on The Internet instead of the demanding interface

Question

I am trying to do an iptables masquerade from a WireGuard Interface and the Internet. It used to work but lately, I did add a few (just four) WireGuard Interfaces, and it stops working for all of them. To confuse me some more, the road warrior connecting directly to the server (with WireGuard) still perform well.

For some reason, the NATing code seems to return the "post-processing UNNATED ANSWER" to the Internet Interface (Call it eth0) instead of wg72. (NOTE: Server address masked with 127.127.127.127)

13:22:26.212282 IP 74.125.5.8.443 > **127.127.127.127.55576**: Flags [S.], seq 4023527294, ack 2220950656, win 65535, options [mss 1408,sackOK,TS val 808558239 ecr 648649,nop,wscale 8], length 0
13:22:26.212299 IP 74.125.5.8.443 > **192.168.61.150.55576**: Flags [S.], seq 4023527294, ack 2220950656, win 65535, options [mss 1408,sackOK,TS val 808558239 ecr 648649,nop,wscale 8], length 0
13:22:26.218685 IP 74.125.5.8.443 > 127.127.127.127.55575: Flags [S.], seq 1487311994, ack 816352590, win 65535, options [mss 1408,sackOK,TS val 808558245 ecr 648649,nop,wscale 8], length 0
13:22:26.218699 IP 74.125.5.8.443 > 192.168.61.150.55575: Flags [S.], seq 1487311994, ack 816352590, win 65535, options [mss 1408,sackOK,TS val 808558245 ecr 648649,nop,wscale 8], length 0

Here is the WireGuard configuration that, I think, include all the pertinent Firewall Rules.

[Interface]
Address = 10.255.16.74/30
Table = off
SaveConfig = true
PostUp = iptables -A FORWARD -i eth0 -o wg72 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
PostUp = iptables -A FORWARD -i eth0 -o wg72 -m conntrack --ctstate INVALID -j ACCEPT
PostUp = iptables -A FORWARD -i wg72 -o eth0 -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 1350
PostUp = iptables -A FORWARD -i wg72 -o eth0  -j ACCEPT
PostUp = iptables -t nat -I POSTROUTING 1 -s 192.168.61.0/24 -o eth0 -j MASQUERADE
PostUp = ip6tables -t nat -I POSTROUTING -o eth0 -j MASQUERADE
PreDown = iptables -D FORWARD -i eth0 -o wg72 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
PreDown = iptables -D FORWARD -i wg72 -o eth0 -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 1350
PreDown = iptables -D FORWARD -i wg72 -o eth0  -j ACCEPT
PreDown = iptables -t nat -D POSTROUTING  -s 192.168.61.0/24 -o eth0 -j MASQUERADE
PreDown = ip6tables -t nat -D POSTROUTING -o eth0 -j MASQUERADE
ListenPort = 12072
PrivateKey = MAYBEMAYBENOT

[Peer]
PublicKey = BLAHBLAHBLAH
AllowedIPs = 0.0.0.0/0
Endpoint = x.x.x.x:18445

And here, the statistic result of trying to use these rules -- I saw that it does not use the specific "RELATED, ESTABLISHED" because the generic rule for the host take care of it already. I have tried many variations in the rules and the result is always the same.

0     0 ACCEPT     all  --  eth0   wg72    0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
0     0 ACCEPT     all  --  eth0   wg72    0.0.0.0/0            0.0.0.0/0            ctstate INVALID
 1953  117K TCPMSS     tcp  --  wg72   eth0    0.0.0.0/0            0.0.0.0/0            tcp flags:0x06/0x02 TCPMSS set 1350
 4334  529K ACCEPT     all  --  wg72   eth0    0.0.0.0/0            0.0.0.0/0           
 6148  925K ACCEPT     all  --  eth0   *       0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED

Here is the NAT Table:

 Chain POSTROUTING (policy ACCEPT 440 packets, 52566 bytes)
 pkts bytes target     prot opt in     out     source               destination         
 2128  208K MASQUERADE  all  --  *      eth0    192.168.61.0/24      0.0.0.0/0 

Here a summary network diagram.

System VPS Running under QENU/KVM I believe)

uname -a : Linux xyz 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19 13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

iptables version: iptables v1.6.1

Answer 1

Found the problems:

-- Did upgrade the Kernel 4.15.0-213-generic. Did solve some instability with Wireguard but more importantly, start to complain about occasional Martian type of routing

-- On my way to find the alien, I did convert from iptables to nft. Was easier to debug and just in case.

-- Finally, the culprit was likely an OSPF flapping route caused by the same subnet use in two areas.