1

I have a conundrum and would like to hear any pointers (been working on this for 2 days now, and need to get this server in to production).

My "google-foo" has not been strong with this one.

I have a multi-homed Linux Server - Centos 6.x

I removed the LACP / bonding on the dual 10 Gig ports on this server.

Despite trying to set up separate ip route's for each NIC, everyhing is going out via NIC em3 (the default route for the system). What I'd like to have this system behaving like is for:

   - em3 to only handle external (internet) traffic.
   - em2 to only handle 10 Gig traffic
   - em1 to handle everything else / internal traffic.

The next hop on the networks are:

  - em1 & em2 go to a good L2/L3, 10 Gig switch 
  - em3 goes to a decent L2/L3, 1 Gig switch

NOTE: IP addresses have been munged a bit.

em1 is an internal, 1 Gig network that has rights to see everything on our internal network as well as the ability to get out to the open internet via our default firewall / router.

em1 is  10.10.18.21/16    gw is to be 10.10.5.1

em2 is an internal, 10 Gig network that is only to see other 10 Gig devices on that network segment.

em2 is  10.16.64.21/26    gw is to be 10.16.64.21

em3 is an external network that allow remote connections to authorized users.

em3 is  10.48.61.80/24    gw is to be 10.48.61.1

I have added entries in the /etc/iproute/rt_tables file as follows:

#
# reserved values
#
255 local
254 main
253 default
0   unspec
#
# local
#
#1  inr.ruhep
10 em1table
20 em2table
30 em3table

I have also created "routes" and "rules" files.

-rw-r--r--  1 root root    98 Feb 15 08:54 route-em1
-rw-r--r--  1 root root    87 Feb 16 12:31 route-em2
-rw-r--r--  1 root root   102 Feb 16 10:37 route-em3
-rw-r--r--  1 root root    65 Feb 15 08:11 rule-em1
-rw-r--r--  1 root root    68 Feb 16 12:32 rule-em2
-rw-r--r--  1 root root    68 Feb 16 10:37 rule-em3


[/etc/sysconfig/network-scripts]$ cat route-em1
10.10.18.0 dev em1 src 10.10.18.21 table em1table
default via 10.10.5.1 dev em1 table em1table
[/etc/sysconfig/network-scripts]$ cat rule-em1
from 10.10.18.21/16 table em1table
to 10.10.5.1 table em1table


[/etc/sysconfig/network-scripts]$ cat route-em2
10.16.64.0/28 dev em2 table em2table
default via 10.16.64.254 dev em2 table em2table
[/etc/sysconfig/network-scripts]$ cat rule-em2
from 10.16.64.21/28 table em2table
to 10.16.64.254 table em2table

67.134
[/etc/sysconfig/network-scripts]$ cat route-em3
10.48.161.0 dev em3 src 10.48.161.82 table em3table
default via 10.48.161.1 dev em3 table em3table
[/etc/sysconfig/network-scripts]$ cat rule-em3
from 10.48.161.82/24 table em3table
to 10.48.161.1 table em3table


[/etc/sysconfig/network-scripts]$ ip route show table em1table
10.10.18.0 dev em1  scope link  src 10.10.18.21 
default via 10.10.5.1 dev em1 

[/etc/sysconfig/network-scripts]$ ip route show table em2table
10.16.64.0/28 dev em2  scope link 
default via 10.16.64.254 dev em2 

[/etc/sysconfig/network-scripts]$ ip route show table em3table
10.48.161.0 dev em3  scope link  src 10.48.161.82 
default via 10.48.161.1 dev em3 

Here is some additional information:

[/etc/sysconfig/network-scripts]$ route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
10.48.161.0     0.0.0.0         255.255.255.0   U     0      0        0 em3
10.16.64.0      0.0.0.0         255.255.192.0   U     0      0        0 em2
10.10.0.0       0.0.0.0         255.255.0.0     U     0      0        0 em1
169.254.0.0     0.0.0.0         255.255.0.0     U     1002   0        0 em1
169.254.0.0     0.0.0.0         255.255.0.0     U     1003   0        0 em2
169.254.0.0     0.0.0.0         255.255.0.0     U     1004   0        0 em3
0.0.0.0         10.48.161.1     0.0.0.0         UG    0      0        0 em3
[/etc/sysconfig/network-scripts]$ 


[/etc/sysconfig/network-scripts]$ ip route show
10.48.161.0/24 dev em3  proto kernel  scope link  src 10.48.161.82 
10.16.64.0/18 dev em2  proto kernel  scope link  src 10.16.64.21 
10.10.0.0/16 dev em1  proto kernel  scope link  src 10.10.18.21 
169.254.0.0/16 dev em1  scope link  metric 1002 
169.254.0.0/16 dev em2  scope link  metric 1003 
169.254.0.0/16 dev em3  scope link  metric 1004 
default via 10.48.161.1 dev em3 
[/etc/sysconfig/network-scripts]$ 


[/etc/sysconfig/network-scripts]$ ip route show to match 10.16.64.0/28
10.16.64.0/18 dev em2  proto kernel  scope link  src 10.16.64.21 
default via 67.134.161.1 dev em3 

[/etc/sysconfig/network-scripts]$ ip route show to match 10.10.18.0/16
10.10.0.0/16 dev em1  proto kernel  scope link  src 10.10.18.21 
default via 67.134.161.1 dev em3 

[/etc/sysconfig/network-scripts]$ ip route show to match 67.134.161.0/24
10.48.161.0/24 dev em3  proto kernel  scope link  src 10.48.161.82 
default via 10.48.161.1 dev em3 



[/etc/sysconfig/network-scripts]$ cat ifcfg-em1
DEVICE=em1
#MASTER=bond0
#SLAVE=yes
HWADDR=c8:1f:66:f4:ce:10
TYPE=Ethernet
UUID=bfa14e4a-66b0-4b83-93a4-094f9090aea7
ONBOOT=yes
NM_CONTROLLED=no
BOOTPROTO=none
IPADDR=10.10.18.21
PREFIX=16
#GATEWAY=10.10.5.1
DNS1=10.10.5.8
DNS2=10.10.5.9
DOMAIN=AXS
DEFROUTE=YES
IP4_FAILURE_FATAL=yes
IP6INIT=no
NAME="em1"
NETMASK=255.255.0.0
IPV6INIT=no
USERCTL=no

[/etc/sysconfig/network-scripts]$ cat ifcfg-em2
DEVICE=em2
#MASTER=bond0
#SLAVE=yes
HWADDR=c8:1f:66:f4:ce:12
TYPE=Ethernet
UUID=c8c5e1fb-ba40-4537-89ad-f7df5de59f8b
ONBOOT=yes
NM_CONTROLLED=no
BOOTPROTO=none
IPADDR=10.16.64.21
PREFIX=18
#GATEWAY=10.16.64.254
DNS1=10.10.5.8
DNS2=10.10.5.9
DOMAIN=AXS
#DEFROUTE=YES
IP4_FAILURE_FATAL=yes
IP6INIT=no
NAME="em2"
#IPADDR=10.16.64.21
NETMASK=255.255.192.0
IPV6INIT=no
USERCTL=no

[/etc/sysconfig/network-scripts]$ cat ifcfg-em3
DEVICE=em3
HWADDR=C8:1F:66:F4:CE:14
TYPE=Ethernet
UUID=aa9552be-0075-46b2-8eff-b7c49c8c999f
ONBOOT=yes
NM_CONTROLLED=no
BOOTPROTO=none
IPADDR=10.48.161.82
PREFIX=24
GATEWAY=10.48.161.1
DNS1=10.10.5.8
DNS2=10.10.5.9
DNS3=205.171.3.65
DNS4=8.8.8.8
DOMAIN=axs.tv
DEFROUTE=yes
IPV4_FAILURE_FATAL=yes
IPV6INIT=no
USERCTL=no

And, when I try to ping on each interface:

[/etc/sysconfig/network-scripts]$ ping -I em1 10.10.5.1
PING 10.10.5.1 (10.10.5.1) from 10.10.18.21 em1: 56(84) bytes of data.
--- 10.10.5.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2846ms
rtt min/avg/max/mdev = 0.081/0.101/0.135/0.024 ms

[/etc/sysconfig/network-scripts]$ ping -I em2 10.16.64.154
PING 10.16.64.154 (10.16.64.154) from 10.16.64.21 em2: 56(84) bytes of data.
--- 10.16.64.154 ping statistics ---
5 packets transmitted, 0 received, +3 errors, 100% packet loss, time 4022ms
pipe 3

[/etc/sysconfig/network-scripts]$ ping -I em3 www.google.com
--- www.google.com ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3366ms
rtt min/avg/max/mdev = 2.657/2.679/2.711/0.041 ms

1 Answer 1

0

I know this is old entry, however, still showup on a multihome search.
I was lost when adding additional gateways and only be able to use one gateway at a time.
Designated primary gateway, was always grabbing all the trafic.
I fix my issue with just three settings:
On your favorite sysctl script add the following: (I use /usr/lib/sysctl.d/99-multihome on Centos)

# add forwarding
net.ipv4.ip_forward = 1
# Accept source routing
net.ipv4.conf.default.accept_source_route = 1
net.ipv4.conf.all.accept_source_route = 1

note, may sure your firewall is up to it's job whan dealing with this change.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .