9

I have a PC(kernel 3.2.0-23-generic) which has 192.168.1.2/24 configured to eth0 interface and also uses 192.168.1.1 and 192.168.1.2 addresses for tun0 interface:

root@T42:~# ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:16:41:54:01:93 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.2/24 scope global eth0
    inet6 fe80::216:41ff:fe54:193/64 scope link
       valid_lft forever preferred_lft forever
3: bond0: <BROADCAST,MULTICAST,MASTER> mtu 1500 qdisc noop state DOWN
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
4: irda0: <NOARP> mtu 2048 qdisc noop state DOWN qlen 8
    link/irda 00:00:00:00 brd ff:ff:ff:ff
5: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:13:ce:8b:99:3e brd ff:ff:ff:ff:ff:ff
    inet 10.30.51.53/24 brd 10.30.51.255 scope global eth1
    inet6 fe80::213:ceff:fe8b:993e/64 scope link
       valid_lft forever preferred_lft forever
6: tun0: <POINTOPOINT,MULTICAST,NOARP> mtu 1500 qdisc pfifo_fast state DOWN qlen 100
    link/none
    inet 192.168.1.1 peer 192.168.1.2/32 scope global tun0
root@T42:~# ip route show dev eth0
192.168.1.0/24  proto kernel  scope link  src 192.168.1.2 
root@T42:~# 

As seen above, tun0 is administratively disabled(ip link set dev tun0 down). Now when I receive ARP requests for 192.168.1.2, the PC does not reply to those requests:

root@T42:~# tcpdump -nei eth0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
15:30:34.875427 00:1a:e2:ae:cb:b7 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.1.2 tell 192.168.1.1, length 46
15:30:36.875268 00:1a:e2:ae:cb:b7 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.1.2 tell 192.168.1.1, length 46
15:30:39.138651 00:1a:e2:ae:cb:b7 > 00:1a:e2:ae:cb:b7, ethertype Loopback (0x9000), length 60:
^C
3 packets captured
3 packets received by filter
0 packets dropped by kernel
root@T42:~#

Only after I delete the tun0 interface(ip link del dev tun0) the PC will reply to ARP request for 192.168.1.2 on eth0 interface.

Routing table looks exactly alike before and after ip link del dev tun0:

root@T42:~# netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         10.30.51.254    0.0.0.0         UG        0 0          0 eth1
10.30.51.0      0.0.0.0         255.255.255.0   U         0 0          0 eth1
192.168.1.0     192.168.1.2     255.255.255.0   UG        0 0          0 eth0
root@T42:~# ip link del dev tun0
root@T42:~# netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         10.30.51.254    0.0.0.0         UG        0 0          0 eth1
10.30.51.0      0.0.0.0         255.255.255.0   U         0 0          0 eth1
192.168.1.0     192.168.1.2     255.255.255.0   UG        0 0          0 eth0
root@T42:~# 

Routing entry below is removed already with ip link set dev tun0 down command:

Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
192.168.1.2     0.0.0.0         255.255.255.255 UH        0 0          0 tun0

However, while routing tables are exactly alike before and after the ip link del dev tun0 command, the actual routing decisions kernel will make are not:

T42:~# ip route get 192.168.1.1
local 192.168.1.1 dev lo  src 192.168.1.1 
    cache <local> 
T42:~# ip link del dev tun0
T42:~# ip route get 192.168.1.1
192.168.1.1 dev eth0  src 192.168.1.2 
    cache  ipid 0x8390
T42:~# 

Is this an expected behavior? Why does kernel ignore the routing table?

4
  • Can you paste the output of netstat -rn for both cases? The routing table is typically the first place to look for these sort of errors. – Clarus May 26 '15 at 20:15
  • @Claris I updated my initial post. – Martin May 27 '15 at 8:40
  • Having the same IP on two interfaces can create issues and is best avoided, that being said you should be able to trace down the issue. The next step is to look at the arp cache, does arp -a show anything useful? – Clarus May 27 '15 at 16:25
  • @Claris Looks like the root cause is that kernel ignores the routing table when tun0 interface is disabled, but present. See the output of ip route get commands in my updated initial post. However, why does kernel behave like that? – Martin May 28 '15 at 8:22
20
+50

Your routing table isn't being ignored, exactly. It's being overruled by a higher-priority routing table.

What's Going On

The routing table you see when you type ip route show isn't the only routing table the kernel uses. In fact, there are three routing tables by default, and they are searched in the order shown by the ip rule command:

# ip rule show
0:      from all lookup local
32766:  from all lookup main
32767:  from all lookup default

The table you're most familiar with is main, but the highest-priority routing table is local. This table is managed by the kernel to keep track of local and broadcast routes: in other words, the local table tells the kernel how to route to the addresses of its own interfaces. It looks something like this:

# ip route show table local
broadcast 127.0.0.0 dev lo  proto kernel  scope link  src 127.0.0.1
local 127.0.0.0/8 dev lo  proto kernel  scope host  src 127.0.0.1
local 127.0.0.1 dev lo  proto kernel  scope host  src 127.0.0.1
broadcast 127.255.255.255 dev lo  proto kernel  scope link  src 127.0.0.1
broadcast 192.168.1.0 dev eth0  proto kernel  scope link  src 192.168.1.2
local 192.168.1.1 dev tun0  proto kernel  scope host  src 192.168.1.1
local 192.168.1.2 dev eth0  proto kernel  scope host  src 192.168.1.2
broadcast 192.168.1.255 dev eth0  proto kernel  scope link  src 192.168.1.2

Check out that line referencing tun0. That's what's causing your strange results from route get. It says 192.168.1.1 is a local address, which means if we want to send an ARP reply to 192.168.1.1, it's easy; we send it to ourself. And since we found a route in the local table, we stop searching for a route, and don't bother checking the main or default tables.

Why multiple tables?

At a minimum, it's nice to be able to type ip route and not see all those "obvious" routes cluttering the display (try typing route print on a Windows machine). It can also serve as some minimal protection against misconfiguration: even if the main routing table has gotten mixed up, the kernel still knows how to talk to itself.

(Why keep local routes in the first place? So the kernel can use the same lookup code for local addresses as it does for everything else. It makes things simpler internally.)

There are other interesting things you can do with this multiple-table scheme. In particular, you can add your own tables, and specify rules for when they are searched. This is called "policy routing", and if you've ever wanted to route a packet based on its source address, this is how to do it in Linux.

If you're doing especially tricky or experimental things, you can add or remove local routes yourself by specifying table local in the ip route command. Unless you know what you're doing, though, you're likely to confuse the kernel. And of course, the kernel will still continue to add and remove its own routes, so you have to watch to make sure yours don't get overwritten.

Finally, if you want to see all of the routing tables at once:

# ip route show table all

For more info, check out the ip-rule(8) man page or the iproute2 docs. You might also try the Advanced Routing and Traffic Control HOWTO for some examples of what you can do.

3
  • Thanks! After ip link set dev tun0 down the local 192.168.1.1 dev tun0 proto kernel scope host src 192.168.1.1 rule was indeed still present in local routing-table. Once I executed ip link del dev tun0 the mentioned rule was removed. Still, one question- am I correct that all modern Linux kernels(2.6.x, 3.x, 4.x) use RPDB for route lookups and thus multiple tables? – Martin Jun 4 '15 at 10:52
  • 2
    Yes, you're correct and more. RPDB is surprisingly old! "The RPDB itself was an integral part of the rewrite of the networking stack in Linux kernel 2.2." And from ip(8): "ip was written by Alexey N. Kuznetsof and added in Linux 2.2." – Jander Jun 4 '15 at 15:19
  • This is one of the best explanations of the multiple kernel routing tables I've seen. Thanks! – djluko Nov 14 '19 at 20:49
1

Your reverse path filtering config is probably the problem. RFC3704 - section 2.4

In Enterprise Linux distributions (RHEL, CentOS, Scientific Linux, et al) the likely best way to resolve this is to modify /etc/sysctl.conf with rp_filter = 2

When RHEL has multiple IPs configured, only one is reachable from a remote network. Or why does RHEL ignore packets when the route for outbound traffic differs from the route of incoming traffic?

1
  • If I use loose RPF check(2) or even disable RPF check(0) altogether with for rp_filter_file in /proc/sys/net/ipv4/conf/*/rp_filter; do echo 0 > "$rp_filter_file"; done the kernel does not use the eth0 interface for routing packets to 192.168.1.1. Only once I delete the tun0 interface with ip link del dev tun0 the kernel starts using the eth0 interface. – Martin Jun 3 '15 at 9:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.