1

(This is the original question but my eventual solution was too different from how I originally asked the question so I'm posting as a new question.)

Background:

I have a VM with three different interfaces: Management (eth0), Production (eth1), and Backup (eth2). Management is for RDP and SSH traffic. Production is for the bulk of traffic. Backup is for communication with backup services, all on the same subnet. Each network is restricted to its own purpose and any traffic that looks like it should be carried by one of the other networks will be dropped (for example, tcp/22 is dropped on the Production network).

Problem:

SSH traffic is coming into the server on the Management port but is leaving on the Production interface:

syslog output from doing a -j LOG shows the output interface is eth1 (production)

This causes the traffic to be dropped since the packets will be tcp/22 going out to the production network.

If I change the default route to point to the management gateway the packets go out the correct interface so ssh works but then all production traffic stops working. I can leave the default route pointing at PROD and it will still work as long as I'm connecting on from a machine on the same subnet.

I need the kernel to set the default gateway dependent on what interface the traffic originated on.

1

Any advanced routing such as setting multiple default gateways will involve policy based routing.

There are multiple solutions that have been posted for forwarding based on marking packets destined for a particular port but the solution I found on Red Hat's customer portal turned out to be much easier and will likely be more widely applicable.

Adding to /etc/rc.local:

  1. Set up the alternate routing table, I've called mine 2.

    /sbin/ip route add table 2 default via <management-gateway>

  2. Set traffic from the management IP to use the alternative routing table with the management gateway set.

    /sbin/ip rule add from <management-IP-address> table 2

The above works by leaving my primary gateway (in the default routing table) pointing out to production (eth1). The gateway in step 2 will be chosen for any traffic originating from the IP address associated with eth0.

In my particular case, the backup network isn't sensitive to a gateway setting since all traffic is on the same subnet. If the backup network did have multiple subnets, though, I could set up a third routing table and do a similar ip rule add for that one.

Without policy-based routing, the kernel would consult the default routing table for deciding the interface to push traffic out. This means the correct interface will be selected if the other node is on the same subnet as one of the interfaces that's connected, otherwise traffic will be sent via the default route. This behavior wouldn't be visible if the network continued to route the packet despite coming out the "wrong" interface but in my case resulted in the traffic being dropped.

After doing policy-based routing, both Production and Management/SSH traffic began functioning as expected.

| improve this answer | |
  • 1
    Step 2 is probably not even needed, that would be for routing packets that are coming through eth0 but not destined for this machine, which is not relevant AFAICS. – wurtel Dec 4 '14 at 15:04
  • You're too fast for me, the same thing occurred to me when I revised it so I just tested it and it isn't needed. It's in the article and so i just kind of did it all verbatim. – Bratchley Dec 4 '14 at 15:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.