1
root@host-3:~# uname -a
Linux host-3 4.4.35-1-pve #1 SMP Fri Dec 9 11:09:55 CET 2016 x86_64 GNU/Linux
root@host-3:~# cat /etc/debian_version
8.9
root@host-3:~# ifconfig
eth0      Link encap:Ethernet  HWaddr xx:xx:xx:xx:xx:xx
          inet addr:192.168.1.2  Bcast:192.168.1.2.255  Mask:255.255.255.0
          inet6 addr: xxxx::xxxx:xxxx:xxxx:xxxx/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:3313 errors:0 dropped:0 overruns:0 frame:0
          TX packets:348 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:224843 (219.5 KiB)  TX bytes:29794 (29.0 KiB)

eth1      Link encap:Ethernet  HWaddr xx:xx:xx:xx:xx:xx
          inet addr:192.168.2.2  Bcast:192.168.2.255  Mask:255.255.255.0
          inet6 addr: xxxx::xxxx:xxxx:xxxx:xxxx/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:3028 errors:0 dropped:0 overruns:0 frame:0
          TX packets:15 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:196588 (191.9 KiB)  TX bytes:1330 (1.2 KiB)

In the setup below, on host-3, traffic does not go out onto the wire. The stack simply sends it back up to the receiving application.

A packet capture on both interfaces shows that no packets ever actually reach any network interface.

Why is this?

2 HOSTS, 1 NIC / HOST: THIS WORKS (FOR COMPARISON TO host-3 BELOW)

---------------------          -------------------------------------------          ---------------------
| Linux Host host-1 |          |        Device Under Test (router)       |          | Linux Host host-2 |
|     192.168.1.2/24|----------|192.168.1.1/24             192.168.2.1/24|----------|192.168.2.2/24     |
---------------------          -------------------------------------------          ---------------------

1 HOST WITH 2 NICs: THIS DOESN'T WORK

---------------------
| Linux Host host-3 |
|     192.168.1.2/24|--------|
|               eth0|        |
|                   |        |
|                   |        |
|               eth1|        |
|     192.168.2.2/24|---|    |
---------------------   |    |
                        |    |
                        |    |
                        |    |
                        |    |         -------------------------------------------
                        |    |         |        Device Under Test (router)       |
                        |    |---------|192.168.1.1/24             192.168.2.1/24|---------|
                        |              -------------------------------------------         |
                        |                                                                  |
                        |                                                                  |
                        |                                                                  |
                        |                                                                  |
                        |------------------------------------------------------------------|

ROUTING TABLE, INCLUDING STATIC ROUTES, FOR host-3

Destination    Gateway        Genmask            Flags Metric Ref Use Iface
192.168.1.0    0.0.0.0        255.255.255.0      U     0      0   0   eth0
192.168.1.2    192.168.2.1    255.255.255.255    UGH   0      0   0   eth1
192.168.2.0    0.0.0.0        255.255.255.0      U     0      0   0   eth1
192.168.2.2    192.168.1.1    255.255.255.255    UGH   0      0   0   eth0

RELEVANT KERNEL PARAMETERS FOR host-3

root@host-3:~# sysctl -a | grep "\.rp_filter"
net.ipv4.conf.eth0.rp_filter = 0
net.ipv4.conf.eth1.rp_filter = 0
net.ipv4.conf.all.rp_filter = 0
net.ipv4.conf.default.rp_filter = 0
net.ipv4.conf.lo.rp_filter = 0

root@host-3:~# sysctl -a | grep "accept_local"
net.ipv4.conf.eth0.accept_local = 1
net.ipv4.conf.eth1.accept_local = 1
net.ipv4.conf.all.accept_local = 1
net.ipv4.conf.default.accept_local = 1
net.ipv4.conf.lo.accept_local = 0

UPDATE #1

In response to the comment querying the type of traffic and whether there are VMs involved...

The traffic is just pings. I have also tested with UDP unicast and found no difference in behavior.

Yes, these are VMs. Actually, more precisely, they are LXC Linux containers running under Proxmox 4.4.

In addition, I've discovered one other thing since my original post.

When pinging, if I use the -I option to specify the egress (i.e. source) IP address, I see no change in behavior. However, if I use ping's -I option to specify the egress interface, things seem to work, with one caveat. I say they work because the ping command gets replies back with round-trip-times of approximately 40 ms. This is about what I'd expect if the packets were actually egressing rather than being short-circuited by the stack (in which case the round-trip-times are sub-millisecond). However, here's the caveat...

Packet captures on both the egress and ingress interfaces show only the echo request. The captures don't show the echo replies. I'm not sure how this can be...

  • 1
    I'm voting to close this question as off-topic because this a doubt about (basic) routing concepts and not Unix – Rui F Ribeiro Aug 22 '17 at 13:07
  • Please provide an authoritative reference that says the behavior I seek is dis-allowed by the RFCs before dismissing me out-of-hand. I have provided authoritative references that say the behavior I seek is compliant. And I have found plenty of Google references stating that others have obtained the behavior I seek on Linux. The question is... Why can I not duplicate their results? – Dave Aug 22 '17 at 13:13
  • As I mentioned in a comment to your answer, this works fine on Windows. It is therefore Linux-specific. This is not a question on basic forwarding concepts. It is much deeper than that, having to do with the details of the implementation of the Linux IP stack. – Dave Aug 22 '17 at 13:15
  • I doubt it a machine not forwarding a packet that is supposed to be local routing being "Linux specific" at all. It is expected and documented behavior. – Rui F Ribeiro Aug 22 '17 at 17:51
1
# ip rule list
0:      from all lookup local 
32766:  from all lookup main 
32767:  from all lookup default

# ip route list table local
[...]
broadcast 192.168.0.0 dev eth0  proto kernel  scope link  src 192.168.0.100
local 192.168.0.100 dev eth0  proto kernel  scope host  src 192.168.0.100
broadcast 192.168.0.255 dev eth0  proto kernel  scope link  src 192.168.0.100

You cannot modify the table local; the kernel maintains it. And this is the table with the highest priority thus it does not matter what you define in the other ones (i.e. main). These entries never match.

  • Hauke, thank you for the response. Do you see network namespaces as a potential solution, or would even that be thwarted by the priority and immutability of the kernel-maintained local table? – Dave Aug 28 '17 at 11:14
  • @Dave Network namespaces do not really change your routing options, they just add more of what you can do without them. You cannot route a local address to a namespace. You may be able to solve your problem by bridging a namespace to eth0+eth1. The bridge does not use routing, it is limited to layer 2 thus these problems should not occur. But you probably need two more namespaces, and you have to create the packets in them. – Hauke Laging Aug 28 '17 at 13:14
1

The traffic won't ever go out of the wire of host 3 , talking to the two different picture subnets, because host3 has NICs with both subnets, and thus it is a multi-homed machine belonging to both of them at the same time. (or it won't go out of the router for what matters, hence you not receiving replies if you force the out interface)

So, what it effectively happens is that any traffic which has a destination both 192.168.1.2/24 and 192.168.2.2/24, is internal traffic to host3, and will go to the loopback of the server, hence you seeing nothing over the wire.

I also recommend having only one default gateway and not two. What happens is that traffic for networks not internal for host3, will be sent to the router/default gateway; that is not the case of the aforementioned netblocks.

I recommend a slight different setup:

---------------------
| Linux Host host-3 |
|     192.168.1.2/24|--------|
|               eth0|        |
|                   |        |
|                   |        |
|               eth1|        |
|     192.168.2.2/24|---|    |
---------------------   |    |
                        |    |
                        |    |
                        |    |
                        |    |         -------------------------------------------
                        |    |         |        Device Under Test (router)       |
                        |    |---------|192.168.3.2/24             192.168.3.1/24|---------|
                        |              -------------------------------------------         |
                        |                                                                  |
                        |                                                                  |
                        |                                                                  |
                        |                                                                  |
                        |------------------------------------------------------------------|

To see traffic over the wire, you will have to send traffic from host3 to the 192.168.3.0/24 network.

  • The behavior I desire is RFC-compliant per RFCs 1122 and 3704. It is, however, not the default for Linux. Hence my use of the kernel parameters rp_filter and accept_local. (As an aside, Windows supports this behavior fine.) Please note that I do not have any default gateways, so nothing is being inadvertently misdirected as a result of such a gateway being present. The recommended setup has the problem of mixing subnets (e.g. 192.168.1.2/24 and 192.168.3.2/24 are directly linked). – Dave Aug 22 '17 at 12:56
  • Routing is routing; unless you mess up with routing priorities, or with the interface doing the output, traffic wont ever leave host3 that has as destination netblocks that are internal to it; ditto for the router. While off-topic, I would recommend a CCNA book. – Rui F Ribeiro Aug 22 '17 at 12:59
  • Please search for accept_local on this page: kernel.org/doc/Documentation/networking/ip-sysctl.txt As far as routing (here, really just forwarding) goes, I have exactly one route with a prefix length match of 32 bits (i.e. a host route). If the stack were even getting to the point of consulting the FIB, this route would have to match. However, the stack doesn't even get to the point of consulting the FIB. It just short-circuits the packet in spite of my setting of sysctl's accept_local kernel parameter. All of my research indicates I should be able to do this. – Dave Aug 22 '17 at 13:10
  • @Dave "From your link..."in combination with suitable routing..."; as I said, you will need to tweak routing priorities. You will also need to use sysctl -w net.ipv4.conf.all.rp_filter=2, which is the tweak I think you are missing, otherwise the packets are discard (anti-spoof measures). – Rui F Ribeiro Aug 22 '17 at 13:50
  • net.ipv4.conf.all.rp_filter=2 is the default. I have tried that value as well. Unfortunately, it did not work. Hence my decision to disable source validation altogether by setting it to 0. – Dave Aug 22 '17 at 13:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.