I am tasked to set up two test servers running CentOS 7.1 (Minimal install; the NetworkManager is disabled on both), and have run into a set of issues that I would like to get help here. The following first describes the background, and then lists the questions towards the end.

Background info

Each one has four network interfaces. Server A resides on subnet A and is connected to switch A. Server B resides on subnet B and is connected to switch B. The two L3 switches A and B are connected to each other as well. Switch A has VLAN0015 that has a as gw for subnet A. Likewise, switch B has a VLAN0016 that has as gw for subnet B. Each switch has a static route for routing traffic between the two VLANs.

The server network setup requirements are the following:

  • All four interfaces on each server should be able to send/receive traffic independently (e.g. ping -I -c 2 (note the two IPs have different last byte!) should see ICMP traffic between the pair only), i.e. any interface in a subnet should be able to communicate with any interface on the other subnet.
  • Interface bonding is intentionally not used
  • All four interfaces should use the gw of their respective subnet (for subnet A:; for subnet B: to communicate with interfaces of the other server on the other subnet.

What I have done:

  • Added to /etc/iproutes/rt_tables the following: 4 ens1f1table 3 ens1f0table 2 ens20f1table 1 ens20f0table
  • Introduced /etc/sysconfig/network-scripts/ route-* and rule-* for each of the four interfaces on each server (an example is given below)

For example, for server A's interface ens20f0, I have the following in /etc/sysconfig/network-scripts/route-ens20f0: dev ens20f0 src table ens20f0table default via dev ens20f0 table ens20f0table

and in its /etc/sysconfig/network-scripts/rule-ens20f0:

from table ens20f0table to table ens20f0table

The setup "sort of works" but from time to time, from one server I couldn't ping any interface of the other server. After some tracerouting, I realized that some interfaces didn't have the right route for traffic. As a get around, I applied the following

/sbin/route add -net gw dev ...

to each of the ens20f0|ens20f1|ens1f0|ens1f1 to force the kernel routing table of e.g. server A to look like below:

Kernel IP routing table Destination Gateway Genmask Flags MSS Window irtt Iface UG 0 0 0 ens10f0 U 0 0 0 ens20f0 U 0 0 0 ens10f0 U 0 0 0 ens20f1 U 0 0 0 ens1f0 U 0 0 0 ens1f1 U 0 0 0 ens10f0 U 0 0 0 ens20f0 U 0 0 0 ens20f1 U 0 0 0 ens1f0 U 0 0 0 ens1f1 UG 0 0 0 ens1f1 UG 0 0 0 ens1f0 UG 0 0 0 ens20f1 UG 0 0 0 ens20f0


Why I tried used the deprecated /sbin/route instead of ip route add? It's because ip route add wouldn't add the desired route entry.

Obviously, the "get-around" setup is not persistent. So,

  1. How can I make my current "get-around" at least persistent across system reboots? There is a /etc/sysconfig/network-scripts/ifup-post, but is it the right place? RHEL 7 documentation doesn't even mention this script, not for non-subscribers anyway.
  2. One server A, I have tried to put in via dev ens20f0 into the route-ens20f0. After a /sbin/ifdown ens20f0 and then /sbin/ifup ens20f0, the desired route didn't show up in the kernel routing table. What did I do wrong? I reviewed man ip-route(8) many times but couldn't tell.
  3. Why the /sbin/route command was able to add a desire route to the kernel routing table, but the newer ip route add|replace couldn't, even on CLI?
  4. The man ip-route mentions an append other than ip route { add | del | change | append | replace } ROUTE, but no descriptions whatsoever for the append directive, what is the use of it?
  5. Is the approach that I described above the correct way of using "policy-based routing" to meet the requirements that are given to me? It seems to "sort of work", e.g. I could ping from some interfaces of a server using ping -I iface to the interfaces of the other server. But the ability to do so seems not to be reliable, as my tracerouting checks have revealed to me.

I am at a loss here as to what else to try. Repeated reading of RHEL 7 Networking Guide 2.4.1. Configuring a Network Interface Using ifcfg Files didn't help. I would be grateful to any hints as to what I have missed.


  • What error message did you receive when you tried to use ip route add? – larsks Aug 27 '15 at 18:09
  • I plan to work on the two systems after the lunch break. I will provide the exact errors here. Thanks for responding. – user183394 Aug 27 '15 at 18:22
  • It just occurred to me (and I will try it after my lunch break once I am in the lab again) perhaps I could just put in a GATEWAY= on all four interfaces of server A, and GATEWAY= for the quad of server B? The routing is done by the two L3 switches. Each server's 4 interfaces are on the same subnet, i.e. the same broadcast domain, thus they don't need routes to reach their respective gw, right? If this approach is correct, then how should these four rule-* files should be modified? Please use the example that I gave above. – user183394 Aug 27 '15 at 18:33
  • Never mind, my last (deleted) comment, misread things. – larsks Aug 27 '15 at 18:34
  • 1
    +1 for not using networkmangler for complex networking. – Criggie Dec 26 '16 at 22:14

A host should not have a gateway configured for an IP network to which it is directly connected.

You do not need multiple routing tables for this - that is massively overcomplicating things, and you have not said anything about putting packet marking into your iptables ruleset, so the routing tables are probably being ignored anyway

Also, I strongly recommend you draw up a picture/map of your network, so it shows the IPs and netmasks. Here's a decent example:

enter image description here

So with a map like this, you can follow the path of a packet and see what happens. Your TCP packets should take the same path there and back, whereas ICMP packets are more forgiving.

Now specific to your config, you have the 192.168.15.x/24 network accessible on all four interfaces ens20f0, ens20f1, ens1f0 and ens1f1.

Your kernel will use the last route that was added. Thing is, the switch will have associated your IP address to one of the earlier added routes, and therefore a different MAC address in the switch's ARP table.

So when it works, it works by accident or coincidence.

I suggest you draw up a plan like the example one, but with your details. And I suspect that you'll have a lightbulb moment while drawing it. Which is the entire point.

If no lightbulbs, then post your image and we'll all discuss it further.


so, you basically are trying to

  1. have four addresses of the same ip-net on the same vlan
  2. route traffic over interfaces with similar ip-adresses on the remote net to the remote interfaces with just these similar addresses?

I've never heard of a solution like yours, but i'm out of the network business for a long time now. So just to give you an idea: Why don't you try to subnet the 15.X / 16.x into 4 subnets on the two machines, and then just add routes to the corresponding net and the def-gw to the subnet interfaces?

With a mask of 26 bits ( this would give 15.0(net) - 15.63(bcast) / 15.64-15.127 / 15.128-15.191 / 15.192-15.255 , similar on the 16.X.

Hope this is of some help...

  • thanks for responding. The four additional subnets approach can't be used. This is because if we need to have any interface on one subnet to be able to communicate with any interface on the other subnet. I will modify the ping example given in the post proper to eliminate such a confusion. – user183394 Aug 27 '15 at 20:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.