How can I make Linux generate different MAC addresses for different bridge devices which are on different PCs?

Question

I have just upgraded two servers from Debian 10 (Buster) to Debian 11 (Bullseye). Afterwards, I could not reach either of them via the network any more. After some investigation, the following problem turned out:

Both machines have a bridge device configured. Obviously, the algorithm which Debian uses to assign MAC addresses to bridge devices has changed from version 10 to 11. After the upgrade, the bridge device on the first server had the same MAC address as the bridge device on the second server, which for sure has not been the case before.

One of the answers there claims that a bridge is a purely internal device and that therefore a bridge's MAC address does not matter. However, this is obviously wrong. At least in my case, packets from both machines were outgoing with the hardware source address being the bridge's MAC address, and the network ports on both machines were processing incoming packets only if they were destined for the bridge's MAC address.

Since that MAC address was the same on both machines, the network became unusable, which is completely logical and understandable.

How can I make Debian generate different MAC addresses for bridge devices which are on different machines (or even on the same machine, but that's currently not my issue)?

Answer 1

Browsing in Internet I found this bug report on systemd-udev related to Debian 11 bridges: systemd-udev interferes with MAC addresses of interfaces it's not supposed to do #21185:

ash.in.ffho.net:~# for n in 0 1 2 3; do ip l add br$n type bridge; done
ash.in.ffho.net:~# ip -br l

br0              DOWN           d2:9e:b3:32:53:42 <BROADCAST,MULTICAST> 
br1              DOWN           e2:00:44:2c:5b:70 <BROADCAST,MULTICAST> 
br2              DOWN           0e:99:b7:42:f0:25 <BROADCAST,MULTICAST> 
br3              DOWN           a6:3f:5f:b5:9a:d6 <BROADCAST,MULTICAST> 
ash.in.ffho.net:~# for n in 0 1 2 3; do ip link del br${n}; done
ash.in.ffho.net:~# for n in 0 1 2 3; do ip l add br$n type bridge; done
ash.in.ffho.net:~# ip -br l

br0              DOWN           d2:9e:b3:32:53:42 <BROADCAST,MULTICAST> 
br1              DOWN           e2:00:44:2c:5b:70 <BROADCAST,MULTICAST> 
br2              DOWN           0e:99:b7:42:f0:25 <BROADCAST,MULTICAST> 
br3              DOWN           a6:3f:5f:b5:9a:d6 <BROADCAST,MULTICAST>

As you can see, the bridges were created with low-level commands, but they always inherit the same MAC address value: a systemd component interferes and sets the MAC address. One can see this in action using ip monitor link:

22: brtest0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default 
    link/ether 0a:ae:c3:0d:ec:68 brd ff:ff:ff:ff:ff:ff
22: brtest0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default 
    link/ether 1a:d0:fc:63:c1:71 brd ff:ff:ff:ff:ff:ff
Deleted 22: brtest0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default 
    link/ether 1a:d0:fc:63:c1:71 brd ff:ff:ff:ff:ff:ff
23: brtest0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default 
    link/ether 4e:e9:11:dd:a5:aa brd ff:ff:ff:ff:ff:ff
23: brtest0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default 
    link/ether 1a:d0:fc:63:c1:71 brd ff:ff:ff:ff:ff:ff

You can see how the MAC address initially random is overwritten to a fixed one, twice to the same value for a given bridge name.

An other side effect is that when interface is set administratively UP, the bridge operational status becomes DOWN instead of UNKNOWN initially because of this (see these answers of mine on SU and SF mentioning behaviors about DOWN and UNKNOWN: How does Linux determine the default MAC address of a bridge device? , linux ipv6 bridge address does not work when mac address is forced). Anyway this doesn't matter anymore once its first bridge port is attached.

Doing the same experiment inside a network namespace (eg: ip add netns experiment and ip netns exec experiment bash -l before running above commands twice) where systemd-udevd does not interfere will show the usual behavior of having different random addresses each time.

This is an effect of systemd ecosystem and doesn't happen on systems not running systemd (or older versions of systemd). One proposed fix is to use:

# /etc/systemd/network/90-bridge.link
[Match]
OriginalName=br*

[Link]
MACAddressPolicy=random

but it appears the real fix is to change the file that participates in generating this "stable random" value, as described there: https://wiki.debian.org/MachineId

Each machine should have a different value. This is especially important for cloned VMs from a base template. The relation between machine-id and the way the bridge "stable" MAC address is generated is mentioned in the patch having implemented the (quite breaking) change:

=== This patch

This patch means that we will set a "stable" MAC for pretty much any virtual device by default, where "stable" means keyed off the machine-id and interface name.

It was also mentioned that this would be having impacts , but this was shrugged off.

This is not limited to interfaces of type bridge but to any interface that would generate a random MAC address: for example types veth, macvlan tuntap are also affected.

I could verify that the same bridge name would get a different "stable random" value after doing the operations described in Debian's link:

rm -f /etc/machine-id /var/lib/dbus/machine-id
dbus-uuidgen --ensure=/etc/machine-id
dbus-uuidgen --ensure

giving now in previous ip monitor a new MAC address for the same bridge name: 32:ee:c8:92:9f:e8 instead of 1a:d0:fc:63:c1:71 when deleting and recreating brtest0.

Deleted 23: brtest0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default 
    link/ether 1a:d0:fc:63:c1:71 brd ff:ff:ff:ff:ff:ff
24: brtest0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default 
    link/ether da:72:b6:63:23:e5 brd ff:ff:ff:ff:ff:ff
24: brtest0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default 
    link/ether 32:ee:c8:92:9f:e8 brd ff:ff:ff:ff:ff:ff

---

Conclusion:

Because the bridge MAC address is now manually set the bridge won't inherit anymore one of the MAC addresses of other interfaces set as bridge ports, including the usual permanent (physical or VM's) interfaces which are expected to have each a different MAC address. Two systems using the same machine-id and the same bridge name (eg: br0) with such bridge participating in routing (ie: there's an IP address configured on the bridge, but even if not the bridge can emit other frames related to bridging depending on its settings) on the same LAN will emit frames with the same source MAC address (bridge's), possibly disrupting switches in the path and anyway ignoring such same source MAC address from the peer.

Answer 2

You can tell Debian to clone the MAC address with a bridge_hw directive.

e.g. my /etc/network/interfaces file:

iface enp2s0 inet manual

auto br0
iface br0 inet dhcp
  bridge_ports enp2s0
  bridge_hw enp2s0

Now both enp2s0 and br0 have the same address.

$ sudo dmesg | grep 18:d6
[    2.660733] r8169 0000:02:00.0 eth0: RTL8168e/8111e, 18:d6:c7:05:89:07, XID 2c2, IRQ 26

$ ifconfig enp2s0 | grep ether ; ifconfig br0 | grep Description:   Debian GNU/Linux 11 (bullseye)
        ether 18:d6:c7:05:89:07  txqueuelen 1000  (Ethernet)
        ether 18:d6:c7:05:89:07  txqueuelen 1000  (Ethernet)

Answer 3

First of all, A.B's answer shows the correct solution. If you are having the same problem, follow A.B's answer to solve it the right way.

However, for the sake of completeness, I'd like to show an alternative (much inferior) solution. I was under pressure and had not expected a correct answer so fast. Therefore I had temporarily solved the problem on my own by slightly changing /etc/network/interfaces:

auto br0
iface br0 inet static
        bridge_ports eno1
        address 192.168.20.11
        netmask 255.255.255.0
        broadcast 192.168.20.255
        gateway 192.168.20.250    
        hwaddress ether 02:01:01:01:11:01

The last line is the key point. It makes the OS assign the indicated MAC address to the bridge. The MAC address shown is an address from the semi-official private MAC address range. I have verified that this MAC address remains the same when you remove or add devices to the bridge.

Of course, you must give each bridge a different MAC address, regardless of whether or not the bridges are on the same machine.

Again, the above method only solves the bridge MAC address problem. I believe that a lot more can go heavily wrong if there are the same machine IDs on different machines. A.B's answer mentions how to create a new machine ID, and a quick test has shown that every machine ID created that way differs from every other machine ID created that way.