3

I have a Linux setup where two macvlan interfaces in mode bridge are added on the same physical interface, in the same IP subnet:

ip link add link eth2 dev mvl0 type macvlan mode bridge
ip link add link eth2 dev mvl1 type macvlan mode bridge
ip addr add 192.168.42.16/24 dev mvl0
ip addr add 192.168.42.17/24 dev mvl1
ip link set dev mvl0 up
ip link set dev mvl1 up

I want to communicate between mvl0 and mvl1 using sockets which are bound to the interfaces, but this doesn't work. For example,

# ping -I mvl0 192.168.42.17

doesn't get any replies. I can see that the kernel tries to do ARP on lo, but that doesn't work as it gets no reply.

  • Is there a way to make this work, for example by manipulating routing or neighbor tables?

  • Should this be considered a bug in the Linux kernel? After all, macvlan interfaces in mode bridge should be able to see each other.

(Background: This is a simulation of two embedded devices running in the same process context. Our framework always binds sockets to interfaces, to ensure that communication really goes through the desired interface. Communication is normally via UDP.)

Improve this question
3
  • Yes, it is the same process.
    – starblue
    Oct 17, 2017 at 20:54
  • 2
    I wanted to suggest to move an interface to an other netns (which would resolve all routing issues), but I can't see how an unique process would deal with this
    – A.B
    Oct 17, 2017 at 21:10
  • 1
    Putting them in different namespaces would work, but this is no solution in the short term as our framework needs to be modified for that. To do this in one process you can temporarily switch the namespace while opening a socket, as the socket stays in the namespace it was opened in.
    – starblue
    Oct 18, 2017 at 7:45

1 Answer 1

Reset to default
3

(Edit: The previous version only worked for outgoing packets (UDP), this version works in both directions (TCP and ping).)

There are various gotchas for a setup like yours: Linux considers an incoming packet with a source address that matches a network interface address a routing error (because under normal circumstances it indicates a routing loop). Also, by default the kernel-maintained local routing table has highest priority, and that will prevent your packets from going out the interface it is bound to.

One can solve the second problem with policy routing: First, delete the overlapping routes (which will only cause trouble):

ip route list
# overlapping routes should look like:
ip route del 192.168.42.0/24 dev mvl0 proto kernel scope link src 192.168.42.16
ip route del 192.168.42.0/24 dev mvl1 proto kernel scope link src 192.168.42.17

Next, give the local table a lower priority (higher numerical value):

ip rule add pref 1000 lookup local
ip rule del pref 0

We need to accept incoming packets by sending them into the local table:

ip rule add pref 100 to 192.168.42.16 iif mvl0 lookup local
ip rule add pref 100 to 192.168.42.17 iif mvl1 lookup local

while all other (outgoing) packets to the resp. destination will use a special table that forces them out in the other interface:

ip rule add pref 200 to 192.168.42.17 lookup 100
ip rule add pref 200 to 192.168.42.16 lookup 101

ip route add default dev mvl0 table 100
ip route add default dev mvl1 table 101

In addition, we have to make take care of the first problem by disabling reverse path filtering (if not already disabled) and enabling packets that have a local source to be accepted:

echo "0" | tee /proc/sys/net/ipv4/conf/mvl{0,1}/rp_filter
echo "1" | tee /proc/sys/net/ipv4/conf/mvl{0,1}/accept_local

Now ping works even if not bound to one of the mvl interfaces. TCP and UDP also work, testing with socat:

socat TCP4-LISTEN:9998,so-bindtodevice=mvl0 -
echo foo | socat - TCP4:192.168.42.16:9998,so-bindtodevice=mvl1

socat UDP4-RECV:9900,so-bindtodevice=mvl0 -
echo foo | socat - UDP4-SENDTO:192.168.42.16:9900,so-bindtodevice=mvl1
Improve this answer
8
  • Many thanks! UDP now works, so that will probably do. I traced the failure of ping to the socket not being in the ping_table in ping.c, for so far unknown reasons, and the last step of delivery to the socket fails. The default for rp_filter is 0 here, accept_local is indeed needed.
    – starblue
    Oct 25, 2017 at 9:37
  • That comment about ping_table was a red herring. (ping.c handles ping sockets, which is a rather exotic feature of the Linux network stack. My ping uses raw sockets.) The actual cause of failure is that ECHO replies arrive on lo, but are expected on mvl0, so they are discarded.
    – starblue
    Oct 29, 2017 at 16:20
  • That should be fixed in the new version, at least it worked on my machine with ping. If you use a different method for ping, please add details, so I can test.
    – dirkt
    Oct 29, 2017 at 18:49
  • Thanks again! It works, but the pref 200 rules look very strange to me, as they send the packet to the opposite interface, not the one having the IP address in the rule. But when I change that it doesn't work, so clearly I don't understand what is going on. More serious is that I don't see how this could scale to three devices, which is likely to be a use case here.
    – starblue
    Nov 6, 2017 at 10:20
  • Yes, the pref 200 rule forces an outgoing packet to use the other interface, instead of the local one which would be normally used. Without that, the packet would just be delivered locally, without going over the bridge. I've to think about how to do it for 3 devices. Don't forget that this method of pretending to have distinct devices by binding to interfaces is fundamentally broken, and we're just applying glue to make it somehow work nevertheless. Switch to namespaces ASAP.
    – dirkt
    Nov 6, 2017 at 14:09

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .