How can I create a virtual test environment for multicast programs?

Question

I am trying to create virtual network devices in order to test and develop multicast programs. My computer has a single NIC with one Ethernet port, which is connected to the Internet. I want a second (virtual) NIC, connected to a bridge, which has 2 other computers connected to it for testing. In other words:

1. Create a virtual NIC.
2. Create a virtual bridge/switch.
3. Connect the virtual NIC to the virtual bridge.
4. Create two additional virtual NIC devices (to be used as remote hosts) and connect them to the virtual bridge.

As I understand it, creating a virtual bridge in Linux implicitly creates and connects a virtual NIC to it, which is accessible as a network interface. I answered a question explaining this here (although I may be wrong).

I know that I could test multicast programs with VMs, but this is quite cumbersome, and my understanding was that with the proper routing tables I should be able to run programs natively if I bind them to the appropriate virtual network device and address. So far, I can't even get pinging to work, much less multicast. This is what I have:

ip link add br0 type bridge
ip link add dum0 type dummy
ip link add dum1 type dummy
ip link set dev dum0 master br0
ip link set dev dum1 master br0
ip addr add 10.0.0.1/24 brd + dev br0
ip addr add 10.0.0.2/24 brd + dev dum0
ip addr add 10.0.0.3/24 brd + dev dum1
ip link set br0 up
ip link set dum0 up
ip link set dum1 up
ip route del 10.0.0.0/24 dev dum0
ip route del 10.0.0.0/24 dev dum1
ip route del broadcast 10.0.0.0 dev dum0
ip route del broadcast 10.0.0.0 dev dum1
ip route del broadcast 10.0.0.255 dev dum0
ip route del broadcast 10.0.0.255 dev dum1
ip route del local 10.0.0.2
ip route del local 10.0.0.3

For convenience, you can use the following to undo that:

ip link del dev dum1
ip link del dev dum0
ip link del dev br0

Upon inspection everything is configured exactly as it would be with physical hardware:

$ ip addr show br0
41: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 56:47:31:fd:10:c0 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.1/24 brd 10.0.0.255 scope global br0
       valid_lft forever preferred_lft forever
    inet6 fe80::5447:31ff:fefd:10c0/64 scope link
       valid_lft forever preferred_lft forever
$ ip addr show dum0
42: dum0: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UNKNOWN group default qlen 1000
    link/ether 56:47:31:fd:10:c0 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.2/24 brd 10.0.0.255 scope global dum0
       valid_lft forever preferred_lft forever
    inet6 fe80::5447:31ff:fefd:10c0/64 scope link
       valid_lft forever preferred_lft forever
$ ip addr show dum1
43: dum1: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UNKNOWN group default qlen 1000
    link/ether d2:47:c8:19:4a:60 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.3/24 brd 10.0.0.255 scope global dum1
       valid_lft forever preferred_lft forever
    inet6 fe80::d047:c8ff:fe19:4a60/64 scope link
       valid_lft forever preferred_lft forever
$ ip route show table main
10.0.0.0/24 dev br0 proto kernel scope link src 10.0.0.1
$ ip route show table local
broadcast 10.0.0.0 dev br0 proto kernel scope link src 10.0.0.1
local 10.0.0.1 dev br0 proto kernel scope host src 10.0.0.1
broadcast 10.0.0.255 dev br0 proto kernel scope link src 10.0.0.1
broadcast 127.0.0.0 dev lo proto kernel scope link src 127.0.0.1
local 127.0.0.0/8 dev lo proto kernel scope host src 127.0.0.1
local 127.0.0.1 dev lo proto kernel scope host src 127.0.0.1
broadcast 127.255.255.255 dev lo proto kernel scope link src 127.0.0.1
$ ip route get to 10.0.0.1
local 10.0.0.1 dev lo src 10.0.0.1 uid 1000
    cache <local>
$ ip route get to 10.0.0.2
10.0.0.2 dev br0 src 10.0.0.1 uid 1000
    cache

... with one exception: the MAC address of dum0 and br0 is the same. This worries me, because it suggests that my understanding of the bridge device is wrong, that it isn't actually a virtual NIC connected to the bridge device, but instead some sort of weird neither-bridge-nor-NIC that can't be used normally. In any case, I don't think this interferes with the rest of the testing. Routing through the dummy devices doesn't work either.

As for the testing, I can only ping any of the devices through the loopback device (lo). The routing table correctly routes packets through br0 to dum0 and dum1, but it returns Destination Host Unreachable:

$ ping -c 2 10.0.0.1 # br0 through lo OK
PING 10.0.0.1 (10.0.0.1) 56(84) bytes of data.
64 bytes from 10.0.0.1: icmp_seq=1 ttl=64 time=0.053 ms
64 bytes from 10.0.0.1: icmp_seq=2 ttl=64 time=0.029 ms

--- 10.0.0.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 56ms
rtt min/avg/max/mdev = 0.029/0.041/0.053/0.012 ms
$ ping -c 2 10.0.0.2 # dum0 through br0 BAD
PING 10.0.0.2 (10.0.0.2) 56(84) bytes of data.
From 10.0.0.1 icmp_seq=1 Destination Host Unreachable
From 10.0.0.1 icmp_seq=2 Destination Host Unreachable

--- 10.0.0.2 ping statistics ---
2 packets transmitted, 0 received, +2 errors, 100% packet loss, time 62ms
pipe 2
$ ping -c 2 -I lo 10.0.0.2 # dum0 through lo OK
ping: Warning: source address might be selected on device other than lo.
PING 10.0.0.2 (10.0.0.2) from x.x.x.x lo: 56(84) bytes of data.
64 bytes from 10.0.0.2: icmp_seq=1 ttl=64 time=0.047 ms
64 bytes from 10.0.0.2: icmp_seq=2 ttl=64 time=0.033 ms

--- 10.0.0.2 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 35ms
rtt min/avg/max/mdev = 0.033/0.040/0.047/0.007 ms

At this point I really don't know what I could be doing wrong. I patched everything through my firewall. I think the only thing are the dummy devices. I tried researching how to "just create a virtual NIC," and that has been extremely frustrating. The ip-link(8) man page lists literally dozens of possible devices, giving absolutely no idea what they do differently from each other or when you would use them. I can't stress enough how hard I tried to research this, because it seems simple enough, but it is very difficult to find information on it (if you don't know so already).

I have read that dummy devices may simply discard data (from one obscure source and nowhere else), in which case maybe they drop ARP requests and I can't find their MAC address (if that is even necessary in this virtual configuration). I also tried using ip tuntap (Linux taps), and that didn't work either, but if I understand those correctly, they provide the raw IP packets (for tunnels) or Ethernet frames (for taps) to programs that request them from the kernel, and otherwise drop all data too.

So, what kind of device do I need? Is this even going to work to test multicast programs? Will I be able to bind to the address of a device, send multicast traffic from it, have it sent across the bridge, and received by a multicast program bound to another device address? This has been quite involved, so I appreciate any help and anyone that can read through it. Thanks!

Answer 1

As suggested by @A.B, the solution is to use multiple network namespaces. We can think of the network stack of a host as a process: Intake -> Processing -> Outtake. Linux does not allow looping the outtake back to the intake, so even though the routing in my original configuration was correct, the packets were dropped -- there is only one network stack, and outtake packets cannot be processed again by the same network stack. Using network namespaces allows the creation of multiple network stacks, which can then respond to ARP requests, pings, and multicast traffic, as desired.

The veth type link can be used to create Ethernet pairs, such that each veth network device represents one end of a link (or perhaps more accurately, a virtual Ethernet network device connected to one end of an Ethernet cable). One end stays in the default network namespace and is added to the virtual bridge, while the other is added to a created network namespace. This allows communication between the namespaces! Here's the code:

ip link add br0 type bridge mcast_snooping 1 mcast_router 2
ip netns add net0
ip link add veth0 type veth peer name veth
ip link set veth netns net0
ip link set dev veth0 master br0
ip netns add net1
ip link add veth1 type veth peer name veth
ip link set veth netns net1
ip link set dev veth1 master br0
ip addr add 10.0.0.1/24 brd + dev br0
ip link set br0 up
ip link set veth0 up
ip link set veth1 up
ip netns exec net0 ip addr add 10.0.0.2/24 brd + dev veth
ip netns exec net1 ip addr add 10.0.0.3/24 brd + dev veth
ip -all netns exec ip link set lo up
ip -all netns exec ip link set veth up

You can use the following to undo that:

ip link del dev veth1
ip link del dev veth0
ip link del dev br0
ip netns del net1
ip netns del net0

This creates a virtual bridge (br0) and two virtual Ethernet pairs (veth0 to veth and veth1 to veth), with the veth devices added to separate network namespaces (before any name conflicts). Here we can see the result:

$ ip addr show br0
25: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 1a:96:25:a0:43:c3 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.1/24 brd 10.0.0.255 scope global br0
       valid_lft forever preferred_lft forever
    inet6 fe80::3c91:4be6:d418:e045/64 scope link 
       valid_lft forever preferred_lft forever
$ ip addr show veth0
27: veth0@if26: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UP group default qlen 1000
    link/ether 1a:96:25:a0:43:c3 brd ff:ff:ff:ff:ff:ff link-netns net0
    inet6 fe80::3c91:4be6:d418:e045/64 scope link 
       valid_lft forever preferred_lft forever
$ ip addr show veth1
29: veth1@if28: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UP group default qlen 1000
    link/ether b6:41:52:5f:ef:eb brd ff:ff:ff:ff:ff:ff link-netns net1
    inet6 fe80::b4fa:8f8c:5976:59c9/64 scope link 
       valid_lft forever preferred_lft forever

Note that the virtual Ethernet devices in the default namespace don't have IP addresses -- they don't need them, because we route through the bridge to reach the hosts. We could give them the IP addresses of their corresponding veth devices in order to route to them directly, without the bridge, if desired. Here's what the created namespaces see:

# ip netns exec net0 ip addr show veth
26: veth@if27: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 46:11:7c:77:fc:01 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.0.0.2/24 brd 10.0.0.255 scope global veth
       valid_lft forever preferred_lft forever
    inet6 fe80::4411:7cff:fe77:fc01/64 scope link 
       valid_lft forever preferred_lft forever
# ip netns exec net1 ip addr show veth
28: veth@if29: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 12:bc:a0:99:8d:43 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.0.0.3/24 brd 10.0.0.255 scope global veth
       valid_lft forever preferred_lft forever
    inet6 fe80::10bc:a0ff:fe99:8d43/64 scope link 
       valid_lft forever preferred_lft forever

Now let's try pinging. We can monitor the ARP cache with ip neighbour and the bridge with tcpdump to give us confidence that things are working as intended:

$ ip neigh
$ ping -c 2 10.0.0.2
PING 10.0.0.2 (10.0.0.2) 56(84) bytes of data.
64 bytes from 10.0.0.2: icmp_seq=1 ttl=64 time=0.124 ms
64 bytes from 10.0.0.2: icmp_seq=2 ttl=64 time=0.059 ms

--- 10.0.0.2 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 58ms
rtt min/avg/max/mdev = 0.059/0.091/0.124/0.033 ms
$ ip neigh
10.0.0.2 dev br0 lladdr 46:11:7c:77:fc:01 REACHABLE

From another terminal, started before the ping:

# tcpdump -i br0
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br0, link-type EN10MB (Ethernet), capture size 262144 bytes
00:54:49.536867 ARP, Request who-has 10.0.0.2 tell 10.0.0.1, length 28
00:54:49.536908 ARP, Reply 10.0.0.2 is-at 46:11:7c:77:fc:01 (oui Unknown), length 28
00:54:49.536911 IP 10.0.0.1 > 10.0.0.2: ICMP echo request, id 9342, seq 1, length 64
00:54:49.536937 IP 10.0.0.2 > 10.0.0.1: ICMP echo reply, id 9342, seq 1, length 64
00:54:50.594136 IP 10.0.0.1 > 10.0.0.2: ICMP echo request, id 9342, seq 2, length 64
00:54:50.594174 IP 10.0.0.2 > 10.0.0.1: ICMP echo reply, id 9342, seq 2, length 64

This can be repeated from within each network namespace with the ip netns exec command, with the same result. Finally, we can test multicast traffic across the two namespaces with a simple socat program listening to a multicast address in one namespace, and sending multicast traffic in the other:

# ip netns exec net0 socat PIPE \
> UDP-RECVFROM:9000,bind=239.0.0.1,ip-add-membership=239.0.0.1:veth &
[1] 9474
# echo ECHO | ip netns exec net1 socat STDIO \
> UDP-DATAGRAM:239.0.0.1:9000,bind=10.0.0.3:9000
ECHO
[1]+  Done

The socat PIPE and UDP-RECVFROM address types wait to receive a UDP datagram on port 9000, write it to an unnamed pipe, read it back from the unnamed pipe, and send it as a unicast UDP datagram on port 9000 back to the source IP address. The STDIO and UDP-DATAGRAM address types read data from stdin, send it as a multicast UDP datagram, receive a unicast UDP datagram, and write its contents to stdout.

From another terminal, started before the server:

# tcpdump -i br0
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br0, link-type EN10MB (Ethernet), capture size 262144 bytes
01:06:04.002116 ARP, Request who-has 10.0.0.3 tell 10.0.0.2, length 28
01:06:04.002129 ARP, Reply 10.0.0.3 is-at 12:bc:a0:99:8d:43 (oui Unknown), length 28
01:06:05.126134 IP 10.0.0.2 > igmp.mcast.net: igmp v3 report, 1 group record(s)
01:06:05.858118 IP 10.0.0.2 > igmp.mcast.net: igmp v3 report, 1 group record(s)
01:06:06.368349 IP 10.0.0.3.9000 > 239.0.0.1.9000: UDP, length 5
01:06:06.368499 IP 10.0.0.2.9000 > 10.0.0.3.9000: UDP, length 5
01:06:06.371106 IP 10.0.0.2 > igmp.mcast.net: igmp v3 report, 1 group record(s)
01:06:06.946105 IP 10.0.0.2 > igmp.mcast.net: igmp v3 report, 1 group record(s)

Incredible.