1

I'm following this video about network namespaces in Linux, and trying to practice it myself, but for some reason it doesn't work. What I've done is:

After creating 2 network namespaces named "blue" and "red":

sudo ip link add bridge type bridge
sudo ip link set dev bridge up
sudo ip link add veth-red type veth peer name veth-red-br
sudo ip link add veth-blue type veth peer name veth-blue-br
sudo ip link set veth-red netns red
sudo ip link set veth-red-br master bridge
sudo ip link set veth-blue netns blue
sudo ip link set veth-blue-br master bridge
sudo ip netns exec red ip addr add 192.168.15.1/24 dev veth-red
sudo ip netns exec blue ip addr add 192.168.15.2/24 dev veth-blue
sudo ip netns exec red ip link set veth-red up
sudo ip link set veth-red-br up
sudo ip netns exec blue ip link set veth-blue up
sudo ip link set veth-blue-br up

Now, if I try to ping from the red to the blue or vice versa, it doesn't work. The weird thing is, that ARP does work, so there is connectivity between the interfaces. I know this because when I run arp in the namespaces I can see the correct values:

$ sudo ip netns exec red arp
Address                  HWtype  HWaddress           Flags Mask            Iface
192.168.15.2             ether   ca:4f:b6:65:a0:f8   C                     veth-red

$ sudo ip netns exec blue ifconfig
veth-blue: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.15.2  netmask 255.255.255.0  broadcast 0.0.0.0
        inet6 fe80::c84f:b6ff:fe65:a0f8  prefixlen 64  scopeid 0x20<link>
        ether ca:4f:b6:65:a0:f8  txqueuelen 1000  (Ethernet)
        RX packets 85  bytes 8946 (8.9 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 27  bytes 1986 (1.9 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

So you can see that the red namespace, in its ARP table, has the MAC address of the blue namespace's veth, so why then ping doesn't work?

1 Answer 1

1

That's because IPv4 got filtered by iptables or nftables. You might think: no way, an Ethernet bridge is working at layer 2, so there's no interaction with iptables working with IPv4 at layer 3.

But there's a way: br_netfilter (showing through the Wayback Machine because they let the certificate expire):

The bridge-netfilter code enables the following functionality:

{Ip,Ip6,Arp}tables can filter bridged IPv4/IPv6/ARP packets, even when encapsulated in an 802.1Q VLAN or PPPoE header. This enables the functionality of a stateful transparent firewall.

All filtering, logging and NAT features of the 3 tools can therefore be used on bridged frames.

See also ebtables/iptables interaction on a Linux-based bridge for details.

So if one does:

iptables -I FORWARD -j DROP
modprobe br_netfilter

(or sets the FORWARD policy to DROP without rules accepting packets) then by default the bridge will drop all IPv4 traffic received at layer 2 through iptables: for IPv4 frames, br_netfilter will temporarily "upgrade" Ethernet frames at layer 2 into IPv4 packets and feed them to iptables, still in the bridge path. Packets dropped by iptables won't go out of the bridge. Those not dropped (none in above's example) are fed back as Ethernet frames into the bridge to continue bridge processing.

Likely culprit: Docker. Docker does enable br_netfilter (probably for inter-container isolation). It's not clearly documented but here's the source of the part of Docker doing it:

  if config.EnableIPTables || config.EnableIP6Tables {
      if _, err := os.Stat("/proc/sys/net/bridge"); err != nil {
          if out, err := exec.Command("modprobe", "-va", "bridge", "br_netfilter").CombinedOutput(); err != nil {
              logrus.Warnf("Running modprobe bridge br_netfilter failed with message: %s, error: %v", out, err)
          }
      }
  }

When it loads the bridge kernel module, it also loads br_netfilter.

and Docker does set iptables' FORWARD policy to DROP:

Docker on a router

Docker also sets the policy for the FORWARD chain to DROP.
[...]

Even if that's not because of Docker, you most certainly have br_netfilter loaded (most likely by physdev then) and firewall rules dropping packets in iptables' (or nftables') FORWARD rules. Else you really have bridge rules, either using ebtables or nftables with the bridge family preventing forwarding frames.

Until kernel 5.3 that was probably the only way to do stateful firewalling at the bridge level.

In this schematic the green (IPv4) boxes in the blue field (Ethernet) represent iptables (or nftables) running in the Ethernet bridge path when br_netfilter enables it:

Packet flow in Netfilter and General Networking

There are ways to do this with bridge granularity rather than system-wide granularity (better do this early in boot so it's inherited in new namespaces too):

  • load br_netfilter but disable its effects:

    modprobe br_netfilter
    sysctl -w net.bridge.bridge-nf-call-arptables=0
    sysctl -w net.bridge.bridge-nf-call-ip6tables=0
    sysctl -w net.bridge.bridge-nf-call-iptables=0
    

    at this point red and blue would be able to communicate, but Docker (likely initial cause of the breakage) would randomly break itself.

  • enable it back on select bridge:

    ip link set bridge type bridge nf_call_iptables 1
    

    communication would stop again between red and blue.

Since kernel 5.3, the system-wide setting got per-network-namespace, should one need to disable (or enable back) this in a non-initial network namespace, affecting all bridges in it:

ip netns FOO sysctl -w net.bridge.bridge-nf-call-iptables=X

Docker doesn't support these modes: it would need to enable the sysctl back on all bridges it created (and possibly on network namespaces it created if there are setups creating bridges there), but doesn't known about this currently.

Of course, removing the kernel module also prevents interactions and would allow traffic back:

rmmod br_netfilter

Still since kernel 5.3, nftables got stateful firewalling support directly in the the bridge family, so using br_netfilter for this feature isn't needed anymore. br_netfilter might still be useful today for firewalling related to PPPoE with the sysctl net.bridge.bridge-nf-filter-pppoe-tagged (it's not enabled by default) for which there's no easy replacement today as far as I know.

See also related answers of mine here and on SF in these Q/A:

5
  • looks like a great answer although didn't get the time to fully learn it. But I don't understand, Docker itself uses the linux bridge functionality, so how is it that Docker disables the use of it?
    – YoavKlein
    Oct 5, 2021 at 13:17
  • My advice: always use Docker separately from any other project requiring network settings. This includes libvirt, LXC and other virtualization technologies.
    – A.B
    Oct 5, 2021 at 13:21
  • don't confuse the bridge functionality which is independent and used everywhere and the "bridge netfilter" functionality which depends on the bridge functionality and can cause troubles with its features. Docker adds rules to manage this functionality. You can do this too, this requiring reading the linked documentation that I put here again: ebtables.netfilter.org/br_fw_ia/br_fw_ia.html . Your question didn't ask how to overcome this, only why it happened. Of course the simpler solution: iptables -I FORWARD -i bridge -j ACCEPT.
    – A.B
    Oct 5, 2021 at 13:25
  • so br_netfilter with addition to the disabling of forwarding in the iptables makes it so that packets won't get from one side of the bridge to the other? why is that?
    – YoavKlein
    Dec 18, 2021 at 18:14
  • another question, how does than docker containers interact through the docker0 bridge?
    – YoavKlein
    Dec 18, 2021 at 18:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.