3

This question already has an answer here:

I have some machines that have several network interfaces, and one or more IP addresses per interfaces. These IP addresses might or might not be in the same subnet. For example, the outcome of ip a might look like that:

1: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000                                                                                                 
    link/ether 11:11:11:11:11:11 brd ff:ff:ff:ff:ff:ff                                                                                                                                        
    inet 10.0.1.2/24 brd 10.0.1.255 scope global eno1                                                                                                                                     
       valid_lft forever preferred_lft forever                                                                                                                                                
    inet 10.12.1.3/24 brd 10.12.1.255 scope global secondary eno1:0                                                                                                                         
       valid_lft forever preferred_lft forever                                                                                                                                                
2: eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000                                                                                                 
    link/ether 22:22:22:22:22:22 brd ff:ff:ff:ff:ff:ff                                                                                                                                        
    inet 10.0.1.4/24 brd 10.0.1.255 scope global eno2                                                                                                                                     
       valid_lft forever preferred_lft forever                                                                                                                                                
    inet 192.168.1.3/24 brd 192.168.1.255 scope global secondary eno2:0                                                                                                                         
       valid_lft forever preferred_lft forever   
2: eno3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000                                                                                                 
    link/ether 33:33:33:33:33:33 brd ff:ff:ff:ff:ff:ff                                                                                                                                        
    inet 172.23.1.2/24 brd 172.23.1.255 scope global eno3                                                                                                                                    
       valid_lft forever preferred_lft forever                              

What I want, is that response to traffic coming into one interface goes out through the same interface; and also be able to tell applications to send traffic through one specific interface, regardless of the destination.

Right now it's not the case; for example, if I ping 10.0.1.2 from another machine in the 192.168.1.0/24 network, the response will be routed through eno2, and that causes all sorts of problems.

How do I do that? Can I set up several routing tables? Can I tag incoming packets with the interface name or something?

marked as duplicate by Christopher, Prvt_Yadv, Fabby, Rui F Ribeiro, Kiwy Apr 29 at 9:48

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • @Christopher had to rewrite my comment when I noticed the usernames where the same ;-) In your solution, if a host in the 192.168.x.x network pings 10.10.10.140, does the response come from the right interface? I couldn't determine that from your answer – Nico Apr 26 at 13:13
  • Okay, thanks then. One last question - I hope - before I close as duplicate. The only difference I see between the debian config in your question and the one in your answer is the number in front of the mgmt table in rt_tables, that went from 1 to 200. Is this important? And thank you for your time :-) – Nico Apr 26 at 13:45
  • Oh okay, but then what's the difference between your question and the answer, i.e. the step that made it work for you? – Nico Apr 26 at 13:53
  • I went to explain, with a reproducible case, how arp_filter is needed, in addition to other routing issues, but it still went as duplicate. – A.B Apr 29 at 21:17
  • @A.B I haven't had time to go back into this yet, I'll try Christopher's solution first (since it seems simpler) and if it doesn't work I'll work to remove the duplicate and try your answer. In any case, thank you very much for taking the time to help me – Nico May 2 at 8:15
0

A remark: your setup requires IP networks 10.0.1.0/24 and 192.168.1.0/24 to be on the same ethernet network. If that's not the case you have to provide detailed network topology, just giving one host wouldn't be enough here. I'm ready to edit this answer to match it, even if the problem is already reproducible and fixable here. That's why in my mockup setup at the end, I'm linking two bridges: to get eno1 and eno2 in the same LAN. I'm also adding a router with IPs 10.0.1.1/24 / 192.168.1.1/24.

Note that I'll mention several times two settings: rp_filter and arp_filter, don't confuse them.

Two different issues with same resolution:

  • ARP flux (note: the two parameters offered in the solution in the link are superseded by the use of arp_filter), existing because by default Linux doesn't really follow routing rules for ARP. This causes problems when addresses on two interfaces are in the same IP LAN. This should be fixed, or you can encounter strange issues coming and going out, lasting a few minutes. So arp_filter must be activated to have it match routing. It would work here without extra effort, because the policy routing to solve the next issue will also solve this one.
  • routing issue, because by default the route to 192.168.0/24 is seen direct via eno2, so eno1 is never considered for reply. When rp_filter is set, the incoming traffic through eno1 is dropped preventing even an answer. If not set or loose, the query to 10.0.1.2 on eno1 gets answered through the wrong interface: eno2.

So to summarize:

  • fixing ARP outgoing behaviour will fix abnormal incoming IP issue, where the incoming request to 10.0.1.2 comes from eno2. This can be seen by bringing down eno1 in the mockup setup below, which after a minute or two (ARP cache expiration on router) will get ping requests to 10.0.1.2 and answers working all using eno2. Now bringing back eno1 up won't revert back incoming IP. That's because the router has still in ARP cache 10.0.1.2 with eno2's MAC address and will refresh by doing unicast queries to that one first, instead of eno1's MAC address. Bringing temporarily down eno2 and flushing ARP on router (or doing DAD/GARP requests using arping on host) would revert to previous situation (no answer if rp_filter=1).

    emphasis mine, from linux/Documentation/networking/ip-sysctl.txt's arp_filter:

    arp_filter - BOOLEAN
    1 - Allows you to have multiple network interfaces on the same
    subnet, and have the ARPs for each interface be answered
    based on whether or not the kernel would route a packet from
    the ARP'd IP
    out that interface (therefore you must use source
    based routing for this to work
    ). In other words it allows control
    of which cards (usually 1) will respond to an arp request.

    0 - (default) The kernel can respond to arp requests with addresses
    from other interfaces
    . This may seem wrong but it usually makes
    sense, because it increases the chance of successful communication.
    IP addresses are owned by the complete host on Linux, not by
    particular interfaces. Only for more complex setups like load-
    balancing, does this behaviour cause problems.

    arp_filter for the interface will be enabled if at least one of
    conf/{all,interface}/arp_filter is set to TRUE,
    it will be disabled otherwise

  • to fix outgoing IP routing issues, you need additional routing tables, one per IP to fix. Each troublesome IP should get a specific rule selecting the IP to use its specific routing table which will not include, thus ignore other local IP's related routes, and instead force the intended interface.

If you intend to deploy such setups, you should probably do some automatization. As this kind of problem usually requires one routing table per IP to fix, and that there can be 2^32 routing tables on modern kernels, you can even map the whole IPv4 to routing tables (considering the few special tables have a number low enough to avoid being mapped and ovewritten, because IPs 0.x.x.x/8 are reserved). I included a few shell functions to ease this. Those routing tables numbers don't need a name but can optionally be named in /etc/iproute2/rt_tables the same way and for the same kind of purpose an IP address can be named in /etc/hosts. Not needed here.

So now it was hopefully explained, here's how to correct:

run the script below on the host.

fix.sh:

#!/bin/sh

map_localip_to_interface () {
    ip -brief address show to "$1" | awk '{ print $1 }'| sed 's/@.*$//'
}

map_ip_to_table () {
        printf '%s\n' "$1" | awk -F. '{ print (((($1*256)+$2)*256)+$3)*256+$4 }'
}

#not used here
map_table_to_ip () {
    local ip
    local n="$1"
    for i in 1 2 3 4; do
        ip="$(($n%256)).$ip"
        n=$(($n/256))
    done
    printf '%s\n' "$ip" | sed 's/\.$//'
}

ip rule add from 10.0.1.2 table $(map_ip_to_table 10.0.1.2)
ip rule add from 10.0.1.4 table $(map_ip_to_table 10.0.1.4)
ip rule add from 192.168.1.3 table $(map_ip_to_table 192.168.1.3)

ip route add table $(map_ip_to_table 10.0.1.2) 10.0.1.0/24 dev $(map_localip_to_interface 10.0.1.2)
ip route add table $(map_ip_to_table 10.0.1.2) default via 10.0.1.1 dev $(map_localip_to_interface 10.0.1.2)
ip route add table $(map_ip_to_table 10.0.1.4) 10.0.1.0/24 dev $(map_localip_to_interface 10.0.1.4)
ip route add table $(map_ip_to_table 10.0.1.4) default via 10.0.1.1 dev $(map_localip_to_interface 10.0.1.4)
ip route add table $(map_ip_to_table 192.168.1.3) 192.168.1.0/24 dev $(map_localip_to_interface 192.168.1.3)
ip route add table $(map_ip_to_table 192.168.1.3) default via 192.168.1.1 dev $(map_localip_to_interface 192.168.1.3)

sysctl -q -w net.ipv4.conf.eno1.arp_filter=1
sysctl -q -w net.ipv4.conf.eno2.arp_filter=1

which will result in:

# ip rule
0:  from all lookup local 
32763:  from 192.168.1.3 lookup 3232235779 
32764:  from 10.0.1.4 lookup 167772420 
32765:  from 10.0.1.2 lookup 167772418 
32766:  from all lookup main 
32767:  from all lookup default 

# ip route show table all |grep 'table [1-9]'
default via 10.0.1.1 dev eno1 table 167772418 
10.0.1.0/24 dev eno1 table 167772418 scope link 
default via 192.168.1.1 dev eno2 table 3232235779 
192.168.1.0/24 dev eno2 table 3232235779 scope link 
default via 10.0.1.1 dev eno2 table 167772420 
10.0.1.0/24 dev eno2 table 167772420 scope link 

# sysctl net.ipv4.conf.eno{1,2}.arp_filter
net.ipv4.conf.eno1.arp_filter = 1
net.ipv4.conf.eno2.arp_filter = 1

All this makes that for any problematic IP, when using it, the routing stack switches to a dedicated routing table which will force the use of the right network interface. ARP will follow the same principle.

Note that the same settings allows locally initiated outgoing IPs to work fine. If 192.168.1.100 is some web server:

ping -I 10.0.1.2 192.168.1.100 # through eno1 and via router
ping -I 10.0.1.4 192.168.1.100 # through eno2 and via router
ping -I 192.168.1.3 192.168.1.100 # through eno2 directly

Likewise if on 192.168.1.100 this command is run with socat (1.7.3.2):

socat tcp4-listen:5555,reuseaddr,fork exec:'printenv SOCAT_PEERADDR'

Then on the multi homed host you can get replies telling all went fine:

# socat tcp4:192.168.1.100:5555 -
192.168.1.3
# socat tcp4:192.168.1.100:5555,bind=10.0.1.2 -
10.0.1.2
# socat tcp4:192.168.1.100:5555,bind=10.0.1.4 -
10.0.1.4

You can verify with tcpdump that the traffic went through the expected interfaces.

If you have more routing issues with other IPs, you should know what to do about it now.

Notes:

  • I left alone 10.12.1.3 here, but since it's on eno1 which now has arp_filter=1, it should probably also receive its own routing table and rule to work properly.
  • If the host is also doing routing, it's possible the rules and routing tables have to be improved. You'll have to test.

Mockup setup:

Run the script setup.sh below as root. Now the issue can be reproduced like this:

term1:

ip netns exec mhost tcpdump -l -e -n -s0 -p -i eno1

term2:

ip netns exec mhost tcpdump -l -e -n -s0 -p -i eno2

term3:

ip netns exec h1921681100 ping -n 10.0.1.2

There won't be any reply. doing at least this:

ip netns exec mhost sysctl -w net.ipv4.conf.eno1.rp_filter=2

would allow assymetric routing, with pings coming from eno1 and leaving to eno2 (or all through eno2 if the "ARP flux" effect kicked in).

Running the script fix.sh above will fix everything:

ip netns exec mhost sh fix.sh

setup.sh:

#!/bin/sh

if ip netns id | grep -qv '^ *$' ; then
        printf 'ERROR: leave netns "%s" first\n' $(ip netns id) >&2
        exit 1
fi

hosts='mhost router h1921681100'
nets='net1001 net1921681 net10121 net172231'

for ns in $hosts $nets; do
    ip netns del $ns 2>/dev/null || :
    ip netns add $ns
    ip netns exec $ns sysctl -q -w net.ipv6.conf.default.disable_ipv6=1
    ip netns exec $ns sysctl -q -w net.ipv4.conf.default.rp_filter=1
    ip netns exec $ns sysctl -q -w net.ipv4.conf.all.rp_filter=1
done

for ns in $hosts; do
    ip -n $ns link set lo up
done

bmac=1
for ns in $nets; do
    ip -n $ns link add bridge0 address 02:00:00:00:00:$(printf '%02d' $bmac) type bridge
    ip -n $ns link set bridge0 up
    bmac=$(($bmac+1))
done

link_lan () {
    [ "$1" = "$2" ] && return 1
    ip -n $2 link add p-$1 type veth peer netns $1 name p-$2 2>/dev/null || return 1
    ip -n $2 link set p-$1 master bridge0 up
    ip -n $1 link set p-$2 master bridge0 up
}

link_lan net1001 net1921681

ip -n h1921681100 link    add eth0 type veth peer netns net1921681 name p-h1921681100
ip -n h1921681100 link    set eth0 up
ip -n  net1921681 link    set p-h1921681100 master bridge0 up
ip -n h1921681100 address add 192.168.1.100/24 broadcast 192.168.1.255 dev eth0
ip -n h1921681100 route   add default via 192.168.1.1

ip netns exec router sysctl -q -w net.ipv4.conf.default.forwarding=1

ip -n router     link    add eth10011 type veth peer netns net1001 name p-router10011
ip -n router     link    set eth10011 up
ip -n net1001    link    set p-router10011 master bridge0 up
ip -n router     address add 10.0.1.1/24 broadcast 10.0.1.255 dev eth10011

ip -n router     link    add eth1921681 type veth peer netns net1921681 name p-router1921681
ip -n router     link    set eth1921681 up
ip -n net1921681 link    set p-router1921681 master bridge0 up
ip -n router     address add 192.168.1.1/24 broadcast 192.168.1.255 dev eth1921681


ip netns exec mhost sysctl -q -w net.ipv4.conf.default.forwarding=0

ip -n mhost      link    add eno1 type veth peer netns net1001 name p-mhosteno1
ip -n mhost      link    set eno1 up
ip -n net1001    link    set p-mhosteno1 master bridge0 up
ip -n mhost      address add 10.0.1.2/24 broadcast 10.0.1.255 dev eno1
ip -n mhost      address add 10.12.1.3/24 broadcast 10.12.1.255 dev eno1

ip -n mhost      link    add eno2 type veth peer netns net1921681 name p-mhosteno2
ip -n mhost      link    set eno2 up
ip -n net1921681 link    set p-mhosteno2 master bridge0 up
ip -n mhost      address add 10.0.1.4/24 broadcast 10.0.1.255 dev eno2
ip -n mhost      address add 192.168.1.3/24 broadcast 192.168.1.255 dev eno2

ip -n mhost      link    add eno3 type veth peer netns net172231 name p-mhosteno3
ip -n mhost      link    set eno3 up
ip -n net172231  link    set p-mhosteno3 master bridge0 up
ip -n mhost      address add 172.23.1.2/24 broadcast 172.23.1.255 dev eno3

Not the answer you're looking for? Browse other questions tagged or ask your own question.