Routing for multiple uplinks/providers

Question

I have a PC with two internet connections. The first one is an Ethernet interface to connect to my local network and access internet via my router. The second one use a GSM modem to connect with ppp.

I would like to be able to use each connection independently when specifying the interface, e.g.:

ping -I eth0 www.google.com
ping -I ppp0 www.google.com

And to be able to use eth0 by default if no interface is selected and the connection via eth0 works but ppp0 if the connection via eth0 does not work.

I read the this article and created the following rules:

# Main table
ip route add 10.0.0.0/24 dev eth0 src 10.0.0.100
ip route add 10.64.64.64 dev ppp0 src 10.123.122.101
ip route add default via 10.0.0.1

# Specific tables
ip route add 10.0.0.0/24 dev eth0 src 10.0.0.100 table eth0
ip route add default via 10.0.0.1 table eth0
ip route add 10.64.64.64 dev ppp0 src 10.123.122.101 table ppp0
ip route add default via 10.64.64.64 table ppp0

# Rules
ip rule add from 10.0.0.100 table eth0
ip rule add from 10.123.122.101 table ppp0

At first it seems to work. However the eth0 table seems to be never used. I expected the following to happen:

ping -I eth0 www.google.ch    # Use default gateway in table eth0
ping -I ppp0 www.google.ch    # Use default gateway in table ppp0
ping www.google.ch            # Use default gateway (main)

However if I remove the default gateway (main) the eth0 interface does not work at all. There is obviously something I don't understand. Can you explain me how to achieve what I want?

EDIT according to @derobert answer:

I tested the following configuration but still get the same error (ok for ppp0 but not working for eth0).

Routes:

# ip rule list
0:      from all lookup local
1500:   from 10.0.0.100 lookup eth0
1501:   from 10.123.122.101 lookup ppp0
2000:   from all fwmark 0x1 lookup eth0
2001:   from all fwmark 0x2 lookup ppp0
32766:  from all lookup main
32767:  from all lookup default

# ip route list table eth0
10.0.0.0/24 dev eth0  src 10.0.0.100
default via 10.0.0.1 dev eth0

# ip route list table ppp0
10.64.64.64 dev ppp0  src 10.123.122.101
default via 10.64.64.64 dev ppp0

# ip route list table main
10.64.64.64 dev ppp0  src 10.123.122.101
192.168.1.0/24 dev eth1  src 192.168.1.1
10.0.0.0/24 dev eth0  src 10.0.0.100

Firewall:

# iptables -L
Chain INPUT (policy DROP)
target     prot opt source               destination
ACCEPT     all  --  anywhere             anywhere

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

# iptables -L -t mangle
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination
CONNMARK   all  --  anywhere             anywhere            CONNMARK restore
RETURN     all  --  anywhere             anywhere            mark match !0x0
MARK       all  --  anywhere             anywhere            MARK set 0x1
MARK       all  --  anywhere             anywhere            MARK set 0x2

Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination
MARK       all  --  anywhere             anywhere            MARK set 0x1
MARK       all  --  anywhere             anywhere            MARK set 0x2
CONNMARK   all  --  anywhere             anywhere            CONNMARK save

# iptables -L -t nat
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination
SNAT       all  --  10.0.0.1             anywhere            to:10.0.0.100
SNAT       all  --  10.64.64.64          anywhere            to:10.123.122.101

I removed all the other firewall rules to be sure they do not interfere. I get this results:

# ip route get 8.2.1.1 from 10.0.0.100
8.2.1.1 from 10.0.0.100 via 10.0.0.1 dev eth0

# ip route get 8.2.1.1 from 10.123.122.101
8.2.1.1 from 10.123.122.101 via 10.64.64.64 dev ppp0

# ping -I ppp0 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=0 ttl=45 time=350.108 ms
64 bytes from 8.8.8.8: seq=1 ttl=45 time=349.768 ms
64 bytes from 8.8.8.8: seq=2 ttl=45 time=329.671 ms
^C
--- 8.8.8.8 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 329.671/343.182/350.108 ms

128# ping -I eth0 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
^C
--- 8.8.8.8 ping statistics ---
5 packets transmitted, 0 packets received, 100% packet loss

The rules seems to be OK but obviously there is another problem. I don't really understand the role of the SNAT here. I added the rules to reflect the configuration of the answer be they might be wrong.

Answer 1

I have a slightly more complicated configuration, which also includes NAT and dynamic (internal) routing on the machine.

There are several parts to it. Let's start with the rules:

Maginot:~# ip rule ls
0:      from all lookup local 
1000:   from all lookup main 
1500:   from 173.167.51.136/29 lookup comcast 
1501:   from 76.160.165.106/29 lookup cavtel 
1502:   from 151.200.251.90/31 lookup vzdsl 
1502:   from 151.200.251.92/31 lookup vzdsl 
1502:   from 151.200.251.94 lookup vzdsl 
2000:   from all fwmark 0x1 lookup comcast 
2001:   from all fwmark 0x2 lookup cavtel 
2002:   from all fwmark 0x3 lookup vzdsl 
2500:   from all lookup comcast 
2501:   from all lookup cavtel 
2502:   from all lookup vzdsl 
32767:  from all lookup default 

(The tables are given names in /etc/iproute2/rt_tables)

As you can see, we have three ISPs at the moment, and we a couple static IP addresses on each. Rules 1500–1502 send the traffic from those source IP addresses out the appropriate interface. Rules 2000–2002 send traffic with a given firewall mark (we'll get to that) out the appropriate interface. Rules 2500–2502 give the preference order of our ISPs, for traffic that hasn't been assigned to an ISP already. If one goes down, you remove the rule for it, and the next one on the list is used.

Each routing table is fairly trivial:

Maginot:~# ip route ls table comcast
default via 173.167.51.142 dev comcast 

(Local and main have a lot more stuff, but that's directly connected and internal routes, respectively. Table default is empty).

Next, its important that once a connection is assigned to an ISP, it stays there (including incoming connections). Trying to move it is impossible, since we've got different IP addresses on each interface, and our ISPs actually have reverse-path filtering. We keep them on the same ISP with firewall rules.

Remember that SNAT is done after routing, so the ip rule policies will not help. You have to use something else (maybe this is your problem?)

iptables -t mangle -A PREROUTING -j CONNMARK --restore-mark
iptables -t mangle -A PREROUTING -m mark ! --mark 0 -j RETURN # if already set, we're done
iptables -t mangle -A PREROUTING -i wan                                       -j MARK --set-mark $MARK_CAVTEL
iptables -t mangle -A PREROUTING -i comcast                                   -j MARK --set-mark $MARK_COMCAST
iptables -t mangle -A PREROUTING -i vz-dsl                                    -j MARK --set-mark $MARK_VZDSL

iptables -t mangle -A POSTROUTING -o wan     -j MARK --set-mark $MARK_CAVTEL
iptables -t mangle -A POSTROUTING -o comcast -j MARK --set-mark $MARK_COMCAST
iptables -t mangle -A POSTROUTING -o vz-dsl  -j MARK --set-mark $MARK_VZDSL
iptables -t mangle -A POSTROUTING -j CONNMARK --save-mark

This is from the shell script that sets it up; the $MARK_… variables match the marks you see in the rules above. Fairly simple: restore the connection mark (remember, marks are per-packet); if there is a mark now (from the connection mark), we're done. Otherwise, set the mark to the one based on the interface involved.

Note it restores the mark before routing, and only saves it afterwards. And the mark is set to the outgoing interface regardless (which is arguable).

Finally, there are the actual NAT rules. We have a several local prefixes, this code is run in a for loop with $local set to each:

iptables -t nat -A POSTROUTING -s $local -o wan     -j SNAT --to-source 76.160.165.106
iptables -t nat -A POSTROUTING -s $local -o comcast -j SNAT --to-source 173.167.51.137
iptables -t nat -A POSTROUTING -s $local -o vz-dsl  -j SNAT --to-source 151.200.251.90

(Note: There are a lot more rules, including some DNAT for DMZ servers, etc... I think I've copied all the relevant ones.)

Answer 2

You will want to set the weight of the eth0 device to be a higher priority than the ppp0 device.

example:

ip route append default scope global nexthop via 10.0.0.1 dev eth0 weight 2 nexthop via 10.64.64.65 dev ppp0 weight 3