route gateway proxy traffic through different interface with identical upstream gateways

Question

My question is similar to the below with a small caveat not answered by any of these threads:

Output traffic on different interfaces based on destination port

https://serverfault.com/questions/648460/load-balancing-network-traffic-using-iptables

iptables --set-mark - Route diferent ports through different interfaces

Only allow certain outbound traffic on certain interfaces

I have three network devices: eth0 (router), eth1 (connected to internet gw1), and eth2 (connected to internet gw2)

eth0 -> eth1 (ip via dhcp) -> gw1 (192.168.0.1 netmask 255.255.255.0) -> internet

eth0 -> eth2 (ip via dhcp) -> gw2 (192.168.0.1 netmask 255.255.255.0) -> internet

I am running Polipo on the router and want traffic of all the people connecting to polipo to go through eth2. Everyone else will have their traffic routed via eth1.

The problem is that both eth1 and eth2 get their IP address via DHCP. And gw1 is identical to gw2. This is part of our infrastructure and they will both have same IP address (i.e. gw1 and gw2 are both 192.168.0.1 with netmask 255.255.255.0).

All answers in the threads mentioned above involve distinct subnets and IP addresses to isolate/mark gateway traffic. In my case this is not an option. I have to accomplish this without changing anything about gw1/gw2 and not involving static IP addresses in eth1 and eth2. Is this even possible?

Answer 1

I'm sorry I can't provide a working example, but what you need should be achievable using netfilter's CONNTRACK target. What differences this from regular MARK is that CONNTRACK marks all the traffic in a connection, while MARK only marks packets. So, the idea would be:

• Mark incoming connections through each interface with different numbers (i.e., "1" for gw1, "2" for gw2)
• Use "ip rule add fwmark 1 lookup 101" and "ip rule add fwmark 2 lookup 102" to make the traffic with mark 1 and mark 2 go to different route tables
• Create a different default route for each table: "ip route add default via 192.168.0.1 dev eth1 table 101" for gw1, and "ip route add default via 192.168.0.1 dev eth2 table 102" for gw2

Connections coming through eth1 would be CONNMARKed with fwmark 1, which would be kept for all packets (incoming and outgoing); the same would happen for connections coming through eth2, with fwmark 2. Thus, the firewall will always know what interface to use in replies. Connections initiated from the firewall or hosts behind it would go through whatever interface/gateway you choose as your primary.

Maybe you can do this more easily using some prepackaged firewall software, like Shorewall. It has multi-ISP support, though I don't know if it uses CONNTRACK or just MARK.