policy routing problem with load balanced ppp connections

Question

This my setup using Debian 6.0.6. It's role is a proxy server and a load balancer using eight 3G modems.

First and foremost:

Whenever a ppp connection dials it takes over the default gateway which is ok but it also cuts me out.The reason it's ok is that in Ubuntu I noticed that it does not do that by default. By default it will leave the ethX gateway untouched.I could have added in the ppp options file replacedefaultroute but problems with iptables in Ubuntu made me switch to Debian.

But now I cannot ping other hosts on the LAN but other hosts can ping me. I also lose connectivity from the outside and the only way I can get access is through another computer from inside the LAN.Also all the hosts in the LAN can use the proxy.

What I found out is that if I add a separate routing table called e1 and copy

192.168.2.0/24 dev eth0  proto kernel  scope link  src 192.168.2.126
default via 192.168.2.3 dev eth0

from the main table into table e1 and then do:

ip rule add table e1

it works, but this is not what I want since all the connections now flow through the eth0 interface.

What can I try to restore the connections comming from outside, and to communicate with the local computers. However all traffic comming from the LAN must still be made through the ppp links.

root@proxy:~# iptables -L OUTPUT -t mangle -v
Chain OUTPUT (policy ACCEPT 433K packets, 217M bytes)
pkts bytes target     prot opt in     out     source               destination
433K  217M CONNMARK   all  --  any    any     anywhere             anywhere            CONNMARK restore
 929 61011 MARK       all  --  any    any     anywhere             anywhere            state NEW statistic mode nth every 8 MARK set 0x1
 929 61721 MARK       all  --  any    any     anywhere             anywhere            state NEW statistic mode nth every 8 packet 1 MARK set 0x2
 929 61461 MARK       all  --  any    any     anywhere             anywhere            state NEW statistic mode nth every 8 packet 2 MARK set 0x3
 929 61438 MARK       all  --  any    any     anywhere             anywhere            state NEW statistic mode nth every 8 packet 3 MARK set 0x4
 929 61530 MARK       all  --  any    any     anywhere             anywhere            state NEW statistic mode nth every 8 packet 4 MARK set 0x5
 929 61022 MARK       all  --  any    any     anywhere             anywhere            state NEW statistic mode nth every 8 packet 5 MARK set 0x6
 929 61738 MARK       all  --  any    any     anywhere             anywhere            state NEW statistic mode nth every 8 packet 6 MARK set 0x7
 928 61224 MARK       all  --  any    any     anywhere             anywhere            state NEW statistic mode nth every 8 packet 7 MARK set 0x8
433K  217M CONNMARK   all  --  any    any     anywhere             anywhere            CONNMARK save


root@proxy:~# iptables -L POSTROUTING -t nat -v
Chain POSTROUTING (policy ACCEPT 285 packets, 18881 bytes)
 pkts bytes target     prot opt in     out     source               destination
  743 49005 MASQUERADE  all  --  any    ppp0    anywhere             anywhere
  705 47291 MASQUERADE  all  --  any    ppp1    anywhere             anywhere
  679 45581 MASQUERADE  all  --  any    ppp2    anywhere             anywhere
  679 45598 MASQUERADE  all  --  any    ppp3    anywhere             anywhere
  670 45177 MASQUERADE  all  --  any    ppp4    anywhere             anywhere
  638 42447 MASQUERADE  all  --  any    ppp5    anywhere             anywhere
  724 48671 MASQUERADE  all  --  any    ppp6    anywhere             anywhere
  679 45182 MASQUERADE  all  --  any    ppp7    anywhere             anywhere
root@proxy:~#

root@proxy:~# ip rule
0:      from all lookup local
32758:  from all fwmark 0x8 lookup d8
32759:  from all fwmark 0x7 lookup d7
32760:  from all fwmark 0x6 lookup d6
32761:  from all fwmark 0x5 lookup d5
32762:  from all fwmark 0x4 lookup d4
32763:  from all fwmark 0x3 lookup d3
32764:  from all fwmark 0x2 lookup d2
32765:  from all fwmark 0x1 lookup d1
32766:  from all lookup main
32767:  from all lookup default

root@proxy:~# ip ro sh t d1
default via 10.64.64.64 dev ppp0
root@proxy:~# ip ro sh t d2
default via 10.64.64.65 dev ppp1
root@proxy:~# ip ro sh t d3
default via 10.64.64.66 dev ppp2
root@proxy:~# ip ro sh t d4
default via 10.64.64.67 dev ppp3
root@proxy:~# ip ro sh t d5
default via 10.64.64.68 dev ppp4
root@proxy:~# ip ro sh t d6
default via 10.64.64.69 dev ppp5
root@proxy:~# ip ro sh t d7
default via 10.64.64.70 dev ppp6
root@proxy:~# ip ro sh t d8
default via 10.64.64.71 dev ppp7


root@proxy:~# ip ro
10.64.64.67 dev ppp3  proto kernel  scope link  src 10.90.33.221
10.64.64.66 dev ppp2  proto kernel  scope link  src 10.18.11.90
10.64.64.65 dev ppp1  proto kernel  scope link  src 10.90.14.235
10.64.64.64 dev ppp0  proto kernel  scope link  src 10.18.27.226
10.64.64.71 dev ppp7  proto kernel  scope link  src 172.22.201.81
10.64.64.70 dev ppp6  proto kernel  scope link  src 10.80.131.6
10.64.64.69 dev ppp5  proto kernel  scope link  src 172.20.17.183
10.64.64.68 dev ppp4  proto kernel  scope link  src 10.80.61.34
192.168.2.0/24 dev eth0  proto kernel  scope link  src 192.168.2.126
default via 192.168.2.3 dev eth0
root@proxy:~#

If you can give me some ideas I would appreciate it.

Thanks!.

Answer 1

1. Access from outside was good when I forwarded the damn SSH port on the router.

2. Communication with other computers in the 192.168.2.0/24 netowrk is ok only after issuing this command in iptables In this instances I only wanted .253 (Local DNS) to talk to me (.126)

   iptables -A OUTPUT -t mangle -s 192.168.2.126 -d 192.168.2.253 -j MARK --set-mark 16

In the iptables list I have rules that basically state, each new connection should be tagged and, perform the routing decision and then go thorugh POSTROUTING chain. Since now I'm tagging packets with specific criteria (tag mark 16) the ip rules will not match the fwmarks so the only route is found in ip rule's default main table

192.168.2.0/24 dev eth0  proto kernel  scope link  src 192.168.2.126  metric 1 

Answer 2

In OUTPUT, you are marking every new packet, even whose destination address is in your LAN.

Your ppp routing tables only have a default entry. That mean every packet ever going to one of these tables will always get routed via your gateway on ppp*. Even if that packet's destination address is in your LAN. That's because the rules are evaluated in increasing priority order, and your d[0-8] tables are tried before main.

As such, pings to your lan get routed to your ppp devices, and get NATed... So if they work, it's because the selected ppp gateway has another host with the same IP address in his routing tables.

I would put every LAN rule in a separate table which is tried before the d[0-8] tables, so that traffic that should go to your lan, goes on your lan.