Linux as router with multiple internet providers

Question

Linux as router: I have 3 Internet providers, each with its own modem.

Provider1, which is gateway address 192.168.1.1
Connected to linux router eth1/192.168.1.2

Provider2, gateway address 192.168.2.1
Connected to linux router eth2/192.168.2.2

Provider3, gateway address 192.168.3.1
Connected to linux router eth3/192.168.3.2

                                                                           ________
                                                   +------------+         /
                                                   |            |        |
                            +----------------------+ Provider 1 +--------|
        __                  |192.168.1.2           |192.168.1.1 |       /
    ___/  \_         +------+-------+              +------------+      |
  _/        \__      |    eth1      |              +------------+      /
 /             \ eth0|              |192.168.2.2   |            |      |
|Client network -----+  ROUTER  eth2|--------------+ Provider 2 +------|     Internet
 \10.0.0.0/24 __/    |              |              |192.168.2.1 |      |
   \__     __/       |    eth3      |              +------------+      \
      \___/          +------+-------+              +------------+       |
                            |192.168.3.2           |            |       \
                            +----------------------+ Provider 3 +-------|
                                                   |192.168.3.1 |       |
                                                   +------------+       \________

I would like to route the clients in network 10.0.0.0/24 by source IP to different gateways.
The interface to the client network is eth0/10.0.0.1, which is the default gateway for all clients.

For example:
10.0.0.11 should be routed to Provider1 @ eth1
10.0.0.12 should be routed to Provider2 @ eth2
...and so on...

I think I need to use ip route and iptables for SNAT, but I have not figured out exactly how.
Here is the script I have so far.
ipv4 forwarding is enabled.

#!/bin/bash
# flush tables
ip route flush table connection1
ip route flush table connection2
ip route flush table connection3

# add the default gateways for each table
ip route add table connection1 default via 192.168.1.1
ip route add table connection2 default via 192.168.2.1
ip route add table connection3 default via 192.168.3.1

# add some IP addresses for marking
iptables -t mangle -A PREROUTING -s 10.0.0.11 -j MARK --set-mark 1
iptables -t mangle -A PREROUTING -s 10.0.0.12 -j MARK --set-mark 2
iptables -t mangle -A PREROUTING -s 10.0.0.13 -j MARK --set-mark 3

# add the source nat rules for each outgoing interface
iptables -t nat -A POSTROUTING -o eth1 -j SNAT --to-source 192.168.1.2
iptables -t nat -A POSTROUTING -o eth2 -j SNAT --to-source 192.168.2.2
iptables -t nat -A POSTROUTING -o eth3 -j SNAT --to-source 192.168.3.2

# link routing tables to connections (?)
ip rule add fwmark 1 table connection1
ip rule add fwmark 2 table connection2
ip rule add fwmark 3 table connection3

#default route for anything not configured above should be eth2

Answer 1

Here is a similar setup from one of our routers (with some irrelevant stuff snipped). Note that this handles incoming connections as well.

Note the use of variables instead of hard-coded mark numbers. So much easier to maintain! They're stored in a separate script, and sourced in. Table names are configured in /etc/iproute2/rt_tables. Interface names are set in /etc/udev/rules.d/70-persistent-net.rules.

##### fwmark ######
iptables -t mangle -F
iptables -t mangle -X

iptables -t mangle -A PREROUTING -j CONNMARK --restore-mark
iptables -t mangle -A PREROUTING -m mark ! --mark 0 -j RETURN # if already set, we're done
iptables -t mangle -A PREROUTING -i wan      -j MARK --set-mark $MARK_CAVTEL
iptables -t mangle -A PREROUTING -i comcast  -j MARK --set-mark $MARK_COMCAST
iptables -t mangle -A PREROUTING -i vz-dsl   -j MARK --set-mark $MARK_VZDSL

iptables -t mangle -A POSTROUTING -o wan     -j MARK --set-mark $MARK_CAVTEL
iptables -t mangle -A POSTROUTING -o comcast -j MARK --set-mark $MARK_COMCAST
iptables -t mangle -A POSTROUTING -o vz-dsl  -j MARK --set-mark $MARK_VZDSL
iptables -t mangle -A POSTROUTING -j CONNMARK --save-mark

##### NAT ######
iptables -t nat -F
iptables -t nat -X
for local in «list of internal IP/netmask combos»; do
    iptables -t nat -A POSTROUTING -s $local -o wan     -j SNAT --to-source «IP»
    iptables -t nat -A POSTROUTING -s $local -o comcast -j SNAT --to-source «IP»
    iptables -t nat -A POSTROUTING -s $local -o vz-dsl  -j SNAT --to-source «IP»
done

# this is an example of what the incoming traffic rules look like
for extip in «list of external IPs»; do
    iptables -t nat -A PREROUTING   -p tcp -d $extip --dport «port» -j DNAT --to-destination «internal-IP»:443
done

And the rules:

ip rule flush
ip rule add from all               pref 1000  lookup main 
ip rule add from A.B.C.D/29        pref 1500  lookup comcast # these IPs are the external ranges (we have multiple IPs on each connection)
ip rule add from E.F.G.H/29        pref 1501  lookup cavtel
ip rule add from I.J.K.L/31        pref 1502  lookup vzdsl
ip rule add from M.N.O.P/31        pref 1502  lookup vzdsl # yes, you can have multiple ranges
ip rule add fwmark $MARK_COMCAST   pref 2000  lookup comcast
ip rule add fwmark $MARK_CAVTEL    pref 2001  lookup cavtel
ip rule add fwmark $MARK_VZDSL     pref 2002  lookup vzdsl
ip rule add                        pref 2500  lookup comcast # the pref order here determines the default—we default to Comcast.
ip rule add                        pref 2501  lookup cavtel
ip rule add                        pref 2502  lookup vzdsl
ip rule add                        pref 32767 lookup default

The routing tables get set up in /etc/network/interfaces, so that taking down an interface makes it switch to using a different one:

iface comcast inet static
        address A.B.C.Q
        netmask 255.255.255.248
        up ip route add table comcast default via A.B.C.R dev comcast
        down ip route flush table comcast

Note: If you're doing filtering as well (which you probably are) you'll also need to add the appropriate rules to FORWARD to ACCEPT the traffic. Especially for any incoming traffic.