2

I have a Rhel 8.7 machine with 2 NICs in different subnets. Let's say eth1-IP:10.10.10.4/24 ,gateway:10.10.10.1. This gateway is also the default gateway for this vm, and a second NIC: eth2, IP:10.10.20.2 , gateway:10.10.20.254.

Situation : This machine already has multiple static routes configured, and my guess is, present in the main routing table, routes I don't want to somehow break.

There is a specific IP (10.10.30.33)that needs to connect via ssh (icmp also is allowed, and used for testing) on both nics. As I have the default gateway on eth1, this IP which is in a different subnet entirely can connect to the eth1 Ip :10.10.10.4 but not on eth2:10.10.20.2 without any modification. When I set up a static route to switch response packets to go through eth2 gateway instead of default gateway, there is reachability but connection is no longer possible on the interface with the default gateway , the connection that before the static route was added worked.

GOAL: Route-only response traffic for specific IP based on the incoming interface. If 10.10.30.33 tries to reach 10.10.10.4 response should be sent on eth1, if it tries 10.10.20.2 it should send on eth2 response packets. Basically, I want both these machine interfaces to be accessible for 10.10.30.33 at the same time.

2
  • I can answer your question on Linux (that will allow you to succeed with ping or ssh etc, but won't survive a reboot), can give a documentation pointer from RedHat on how to integrate the configuration on RHEL 8, but I can't give an answer with such integration, because I don't have enough specific knowledge about RHEL 8. Is that good enough?
    – A.B
    Commented Apr 1, 2023 at 17:59
  • Will definitely provide some (hopefully) new insights.. i have read here about policy based routing , custom routing tables .. but i only need this behaviour for a specific IP only , and for the rest of the static routes present not to be affected anyhow
    – l0cal
    Commented Apr 1, 2023 at 18:21

1 Answer 1

3

TL;DR

Just run these 3 commands to get the multi-homed setup working (only) with 10.10.30.33:

ip route add 10.10.20.0/24 dev eth2 table 1002
ip route add default via 10.10.20.254 dev eth2 table 1002
ip rule add from 10.10.20.2 to 10.10.30.33 iif lo lookup 1002

Detailed explanation

This requires policy routing, because by default Linux will use only the default route with lowest metric from the main routing table. Policy routing allows to use alternate routing tables with alternate default routes depending on routing rule selectors to choose an alternate outcome. In most cases the selector is about the source (rather than the destination: the route is enough for this), but OP explicitly specified that only 10.10.30.33 should trigger a different behavior, so an additional selector about 10.10.30.33 is also used in the routing rules.

To integrate this properly on RHEL 8 using NetworkManager or else with the deprecated method using /etc/sysconfig/network-scripts/ (if NetworkManager is installed I can't tell if the old method works):

For a generic multi-homed use not restricted to 10.10.30.33, just remove in following routing rules any to 10.10.30.33.

  • Replicate only routes that are using eth2, as if eth1 didn't exist in alternate routing table 1002 (value 1002 randomly chosen):

    ip route add 10.10.20.0/24 dev eth2 table 1002
    ip route add default via 10.10.20.254 dev eth2 table 1002
    
  • For locally initiated traffic (including replies) coming from 10.10.20.2 to select the alternate routing table 1002, use the routing rule:

    ip rule add from 10.10.20.2 to 10.10.30.33 iif lo lookup 1002
    

    If it helps for integration, for most cases, iif lo is not needed and can be removed from the rule.

  • The next rule is needed only when these conditions are met:

    • when binding (only) to an interface (eg: ping -I eth2 10.10.30.33) instead of binding to its IP address (ping -I 10.10.20.2 10.10.30.33)
    • when there is no default route (with a higher metric than the selected default route) using this bound interface in the main routing table. Without this route, a default route would be forced to this interface, but without gateway, preventing the correct path to be used (symptom: system would emit ARP to an other network).
    • only relevant for an ICMP client, or an UDP client in unconnected mode (ie not using the connect(2) system call), because in most setups the correct source IP address will be set from the main routing table where the address is set as hinted source if the socket uses connect(2): the previous routing rule is then triggered anyway when emitting.

     

    ip rule add to 10.10.30.33 oif eth2 lookup 1002
    

    As OP did appear to add a 2nd default route:

    and a second NIC: eth2, IP:10.10.20.2 , gateway:10.10.20.254

    then the rule above is not needed, the main routing table is good enough.

    In many setups, there is no default route defined for the alternate interface (eth2) and the rule is then useful.

  • Incoming traffic doesn't need a specific rule: the route lookup for incoming packets is handled by the early local routing table.

    This could be needed for routed traffic, for example if machine was running VMs or containers. OP didn't mention this. Anyway this would require more detailed information and probably involve Netfilter (iptables or nftables) for a router case.

From now on, the system 10.10.30.33 should be able to ssh or ping to both 10.10.10.4 or 10.10.20.2. Likewise (if no firewall prevents it) the dual-homed machine can choose to reach 10.10.30.33 either as usually with the default settings from the main routing table or else by binding to the interface eth2 (eg: ping -I eth2 10.10.30.33 or ssh -B eth2 10.10.30.33) or the address 10.10.20.2 (eg: ping -I 10.10.20.2 10.10.30.33 or ssh -b 10.10.20.2 10.10.30.33).

The equivalent configuration about eth1 is optional because already handled correctly in the main routing table. It would be done the same:

ip route add 10.10.10.0/24 dev eth1 table 1001
ip route add default via 10.10.10.1 dev eth1 table 1001
ip rule add from 10.10.10.4 to 10.10.30.33 iif lo lookup 1001

Caveat: UDP services

By default an UDP reply doesn't inherit the context to know what was the local system's IP address or interface of the incoming query. So by default when an UDP socket is bound to INADDR_ANY (0.0.0.0), 0.0.0.0 will be the initial source address presented to the network stack to resolve the route, the interface and too late the source IP address. That means none of the previous routing rules will match since 10.10.20.2 wasn't presented to these rules: the main interface is still be chosen. Once the interface is known, the adequate IP address for this interface will be chosen, and by default will be the selected interface's primary IP address for the route. That means that an UDP service that is not multi-homed aware will still reply using the wrong interface (eth1) and address (10.10.10.4) when receiving a request to 10.10.20.2 (unless it's from LAN 10.10.20.0/24 where it will work). This behavior might not be limited to Linux but might affect for similar reasons other OSes using the weak host model and the BSD socket API.

Should such UDP service require access from both interfaces, There are two ways to overcome this:

  • bind explicitly to each address, not to INADDR_ANY. Eg should an HTTP/3 service use UDP port 80, instead of binding to 0.0.0.0:80 it should be configured to bind twice with two separate sockets: to 10.10.10.4:80 and 10.10.20.2:80. In such case, the reply through the UDP socket will choose the bound address as source, triggering the correct routing rules and interface. For example that's what is usually doing the DNS server bind9. Most applications have specific configuration for this.

  • have the application be multi-homed aware when handling UDP. This requires the application to activate ancillary messages to retrieve the actual incoming interface and/or address the UDP packet was received on, using the IP_PKTINFO socket option and use this information when replying. For example that's what is usually doing the DNS server unbound. This requires specific code in the application.

TCP does not suffer from this issue: the reply has context to choose the right source address triggering the correct routing rules.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .