5

I have a PC with two internet connections. The first one is an Ethernet interface to connect to my local network and access internet via my router. The second one use a GSM modem to connect with ppp.

I would like to be able to use each connection independently when specifying the interface, e.g.:

ping -I eth0 www.google.com
ping -I ppp0 www.google.com

And to be able to use eth0 by default if no interface is selected and the connection via eth0 works but ppp0 if the connection via eth0 does not work.

I read the this article and created the following rules:

# Main table
ip route add 10.0.0.0/24 dev eth0 src 10.0.0.100
ip route add 10.64.64.64 dev ppp0 src 10.123.122.101
ip route add default via 10.0.0.1

# Specific tables
ip route add 10.0.0.0/24 dev eth0 src 10.0.0.100 table eth0
ip route add default via 10.0.0.1 table eth0
ip route add 10.64.64.64 dev ppp0 src 10.123.122.101 table ppp0
ip route add default via 10.64.64.64 table ppp0

# Rules
ip rule add from 10.0.0.100 table eth0
ip rule add from 10.123.122.101 table ppp0

At first it seems to work. However the eth0 table seems to be never used. I expected the following to happen:

ping -I eth0 www.google.ch    # Use default gateway in table eth0
ping -I ppp0 www.google.ch    # Use default gateway in table ppp0
ping www.google.ch            # Use default gateway (main)

However if I remove the default gateway (main) the eth0 interface does not work at all. There is obviously something I don't understand. Can you explain me how to achieve what I want?

EDIT according to @derobert answer:

I tested the following configuration but still get the same error (ok for ppp0 but not working for eth0).

Routes:

# ip rule list
0:      from all lookup local
1500:   from 10.0.0.100 lookup eth0
1501:   from 10.123.122.101 lookup ppp0
2000:   from all fwmark 0x1 lookup eth0
2001:   from all fwmark 0x2 lookup ppp0
32766:  from all lookup main
32767:  from all lookup default

# ip route list table eth0
10.0.0.0/24 dev eth0  src 10.0.0.100
default via 10.0.0.1 dev eth0

# ip route list table ppp0
10.64.64.64 dev ppp0  src 10.123.122.101
default via 10.64.64.64 dev ppp0

# ip route list table main
10.64.64.64 dev ppp0  src 10.123.122.101
192.168.1.0/24 dev eth1  src 192.168.1.1
10.0.0.0/24 dev eth0  src 10.0.0.100

Firewall:

# iptables -L
Chain INPUT (policy DROP)
target     prot opt source               destination
ACCEPT     all  --  anywhere             anywhere

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

# iptables -L -t mangle
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination
CONNMARK   all  --  anywhere             anywhere            CONNMARK restore
RETURN     all  --  anywhere             anywhere            mark match !0x0
MARK       all  --  anywhere             anywhere            MARK set 0x1
MARK       all  --  anywhere             anywhere            MARK set 0x2

Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination
MARK       all  --  anywhere             anywhere            MARK set 0x1
MARK       all  --  anywhere             anywhere            MARK set 0x2
CONNMARK   all  --  anywhere             anywhere            CONNMARK save

# iptables -L -t nat
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination
SNAT       all  --  10.0.0.1             anywhere            to:10.0.0.100
SNAT       all  --  10.64.64.64          anywhere            to:10.123.122.101

I removed all the other firewall rules to be sure they do not interfere. I get this results:

# ip route get 8.2.1.1 from 10.0.0.100
8.2.1.1 from 10.0.0.100 via 10.0.0.1 dev eth0

# ip route get 8.2.1.1 from 10.123.122.101
8.2.1.1 from 10.123.122.101 via 10.64.64.64 dev ppp0

# ping -I ppp0 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=0 ttl=45 time=350.108 ms
64 bytes from 8.8.8.8: seq=1 ttl=45 time=349.768 ms
64 bytes from 8.8.8.8: seq=2 ttl=45 time=329.671 ms
^C
--- 8.8.8.8 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 329.671/343.182/350.108 ms

128# ping -I eth0 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
^C
--- 8.8.8.8 ping statistics ---
5 packets transmitted, 0 packets received, 100% packet loss

The rules seems to be OK but obviously there is another problem. I don't really understand the role of the SNAT here. I added the rules to reflect the configuration of the answer be they might be wrong.

6
  • The interface-specific routes you are adding are not in themselves default routes - they go to specific networks. For example the first line adds a route for eth0 to the network 10.0.0.0/24. The default line is required as otherwise there is nowhere specified for the packets to go. I've previously done this using 0.0.0.0/0 as the network/mask which means everything. But I'm afraid I'm not clear enough about your situation to offer an answer beyond that's why you need the default line.
    – Dave C
    Mar 28, 2013 at 16:16
  • What do you get from ip -4 rule ls? I don't see any pref in your rules, want to make sure they're in the right order.
    – derobert
    Mar 28, 2013 at 18:33
  • @derobert The result for command ip rule list is the following: 0: from all lookup local 32764: from 10.0.0.100 lookup eth0 32765: from 10.123.122.101 lookup ppp0 32766: from all lookup main 32767: from all lookup default
    – slepasteur
    Apr 8, 2013 at 12:16
  • That looks like it should work. I have a very similar config, and it works. What do you get from ip route get 8.2.1.1 from 10.0.0.100 and ip route get 8.2.1.1 from 10.123.122.101. Also, those are RFC1918 space, not sure if you've just changed the numbers for privacy, or if this is internal traffic, or if there is NAT running on this box?
    – derobert
    Apr 9, 2013 at 16:15
  • See my edited answer for the result of the commands. Indeed I've changed the IP for privacy, that might not have been my best idea :).
    – slepasteur
    Apr 11, 2013 at 15:45

2 Answers 2

Reset to default
3

I have a slightly more complicated configuration, which also includes NAT and dynamic (internal) routing on the machine.

There are several parts to it. Let's start with the rules:

Maginot:~# ip rule ls
0:      from all lookup local 
1000:   from all lookup main 
1500:   from 173.167.51.136/29 lookup comcast 
1501:   from 76.160.165.106/29 lookup cavtel 
1502:   from 151.200.251.90/31 lookup vzdsl 
1502:   from 151.200.251.92/31 lookup vzdsl 
1502:   from 151.200.251.94 lookup vzdsl 
2000:   from all fwmark 0x1 lookup comcast 
2001:   from all fwmark 0x2 lookup cavtel 
2002:   from all fwmark 0x3 lookup vzdsl 
2500:   from all lookup comcast 
2501:   from all lookup cavtel 
2502:   from all lookup vzdsl 
32767:  from all lookup default

(The tables are given names in /etc/iproute2/rt_tables)

As you can see, we have three ISPs at the moment, and we a couple static IP addresses on each. Rules 1500–1502 send the traffic from those source IP addresses out the appropriate interface. Rules 2000–2002 send traffic with a given firewall mark (we'll get to that) out the appropriate interface. Rules 2500–2502 give the preference order of our ISPs, for traffic that hasn't been assigned to an ISP already. If one goes down, you remove the rule for it, and the next one on the list is used.

Each routing table is fairly trivial:

Maginot:~# ip route ls table comcast
default via 173.167.51.142 dev comcast

(Local and main have a lot more stuff, but that's directly connected and internal routes, respectively. Table default is empty).

Next, its important that once a connection is assigned to an ISP, it stays there (including incoming connections). Trying to move it is impossible, since we've got different IP addresses on each interface, and our ISPs actually have reverse-path filtering. We keep them on the same ISP with firewall rules.

Remember that SNAT is done after routing, so the ip rule policies will not help. You have to use something else (maybe this is your problem?)

iptables -t mangle -A PREROUTING -j CONNMARK --restore-mark
iptables -t mangle -A PREROUTING -m mark ! --mark 0 -j RETURN # if already set, we're done
iptables -t mangle -A PREROUTING -i wan                                       -j MARK --set-mark $MARK_CAVTEL
iptables -t mangle -A PREROUTING -i comcast                                   -j MARK --set-mark $MARK_COMCAST
iptables -t mangle -A PREROUTING -i vz-dsl                                    -j MARK --set-mark $MARK_VZDSL

iptables -t mangle -A POSTROUTING -o wan     -j MARK --set-mark $MARK_CAVTEL
iptables -t mangle -A POSTROUTING -o comcast -j MARK --set-mark $MARK_COMCAST
iptables -t mangle -A POSTROUTING -o vz-dsl  -j MARK --set-mark $MARK_VZDSL
iptables -t mangle -A POSTROUTING -j CONNMARK --save-mark

This is from the shell script that sets it up; the $MARK_… variables match the marks you see in the rules above. Fairly simple: restore the connection mark (remember, marks are per-packet); if there is a mark now (from the connection mark), we're done. Otherwise, set the mark to the one based on the interface involved.

Note it restores the mark before routing, and only saves it afterwards. And the mark is set to the outgoing interface regardless (which is arguable).

Finally, there are the actual NAT rules. We have a several local prefixes, this code is run in a for loop with $local set to each:

iptables -t nat -A POSTROUTING -s $local -o wan     -j SNAT --to-source 76.160.165.106
iptables -t nat -A POSTROUTING -s $local -o comcast -j SNAT --to-source 173.167.51.137
iptables -t nat -A POSTROUTING -s $local -o vz-dsl  -j SNAT --to-source 151.200.251.90

(Note: There are a lot more rules, including some DNAT for DMZ servers, etc... I think I've copied all the relevant ones.)

3
  • Thank you very much for the answer. I edited my question after testing your configuration.
    – slepasteur
    Apr 11, 2013 at 15:46
  • @derobert: I would like someday to know so much about Linux networking and TCP/IP, but at the moment barely know how to use iproute2 and setup a bare-bones OpenVPN. What should I read to develop a good basic grounding in Linux networking?
    – TomRoche
    Mar 12, 2015 at 3:05
  • @TomRoche I'd suggest stopping by chat.stackexchange.com/rooms/26/unix-and-linux and asking.
    – derobert
    Mar 12, 2015 at 22:04
1

You will want to set the weight of the eth0 device to be a higher priority than the ppp0 device.

example:

ip route append default scope global nexthop via 10.0.0.1 dev eth0 weight 2 nexthop via 10.64.64.65 dev ppp0 weight 3
1
  • The command gives me the following error: ip: either "to" is duplicate, or "scope" is garbage
    – slepasteur
    Apr 8, 2013 at 12:35

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .