5

I have just upgraded two servers from Debian 10 (Buster) to Debian 11 (Bullseye). Afterwards, I could not reach either of them via the network any more. After some investigation, the following problem turned out:

Both machines have a bridge device configured. Obviously, the algorithm which Debian uses to assign MAC addresses to bridge devices has changed from version 10 to 11. After the upgrade, the bridge device on the first server had the same MAC address as the bridge device on the second server, which for sure has not been the case before.

One of the answers there claims that a bridge is a purely internal device and that therefore a bridge's MAC address does not matter. However, this is obviously wrong. At least in my case, packets from both machines were outgoing with the hardware source address being the bridge's MAC address, and the network ports on both machines were processing incoming packets only if they were destined for the bridge's MAC address.

Since that MAC address was the same on both machines, the network became unusable, which is completely logical and understandable.

How can I make Debian generate different MAC addresses for bridge devices which are on different machines (or even on the same machine, but that's currently not my issue)?

4
  • 1
    How are you building the bridge? On my machine I have /etc/network/interfaces build the bridge with only the primary ethernet adapter connected and the bridge gets the MAC address of the ethernet adapter. It's probably worth updating the question with how you're building it. Oct 1 at 11:54
  • @StephenHarris see the update on my answer about the MAC address not inherited anymore
    – A.B
    Oct 1 at 12:38
  • @StephenHarris Some time ago, I have created the bridge manually on both machines and have assigned one ethernet adapter to it (/dev/eno1 in my case), but there are virtual VM network devices which get dynamically added and removed from it as VMs start and stop. The IP configuration of the bridge is in /etc/network/interfaces (as in your case). That worked so far, except for the problem with the MAC address. In the meantime, I could confirm that A.B's answer is correct.
    – Binarus
    Oct 1 at 15:36
  • 1
    @A.B it may matter how the bridge is created. In /etc/network/interfaces on my Debian 11 machine I have bridge_hw enp2s0 in the br0 definition. That may be what lets it get inherited; % /usr/bin/lsb_release -a | grep Descr; ifconfig enp2s0 | grep ether ; ifconfig br0 | grep ether Description: Debian GNU/Linux 11 (bullseye) ether 18:d6:c7:05:89:07 txqueuelen 1000 (Ethernet) ether 18:d6:c7:05:89:07 txqueuelen 1000 (Ethernet) iface enp2s0 inet manual auto br0 iface br0 inet dhcp bridge_ports enp2s0 bridge_hw enp2s0 Oct 1 at 19:48

3 Answers 3

6

Browsing in Internet I found this bug report on systemd-udev related to Debian 11 bridges: systemd-udev interferes with MAC addresses of interfaces it's not supposed to do #21185:

ash.in.ffho.net:~# for n in 0 1 2 3; do ip l add br$n type bridge; done
ash.in.ffho.net:~# ip -br l

br0              DOWN           d2:9e:b3:32:53:42 <BROADCAST,MULTICAST> 
br1              DOWN           e2:00:44:2c:5b:70 <BROADCAST,MULTICAST> 
br2              DOWN           0e:99:b7:42:f0:25 <BROADCAST,MULTICAST> 
br3              DOWN           a6:3f:5f:b5:9a:d6 <BROADCAST,MULTICAST> 
ash.in.ffho.net:~# for n in 0 1 2 3; do ip link del br${n}; done
ash.in.ffho.net:~# for n in 0 1 2 3; do ip l add br$n type bridge; done
ash.in.ffho.net:~# ip -br l

br0              DOWN           d2:9e:b3:32:53:42 <BROADCAST,MULTICAST> 
br1              DOWN           e2:00:44:2c:5b:70 <BROADCAST,MULTICAST> 
br2              DOWN           0e:99:b7:42:f0:25 <BROADCAST,MULTICAST> 
br3              DOWN           a6:3f:5f:b5:9a:d6 <BROADCAST,MULTICAST>

As you can see, the bridges were created with low-level commands, but they always inherit the same MAC address value: a systemd component interferes and sets the MAC address. One can see this in action using ip monitor link:

22: brtest0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default 
    link/ether 0a:ae:c3:0d:ec:68 brd ff:ff:ff:ff:ff:ff
22: brtest0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default 
    link/ether 1a:d0:fc:63:c1:71 brd ff:ff:ff:ff:ff:ff
Deleted 22: brtest0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default 
    link/ether 1a:d0:fc:63:c1:71 brd ff:ff:ff:ff:ff:ff
23: brtest0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default 
    link/ether 4e:e9:11:dd:a5:aa brd ff:ff:ff:ff:ff:ff
23: brtest0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default 
    link/ether 1a:d0:fc:63:c1:71 brd ff:ff:ff:ff:ff:ff

You can see how the MAC address initially random is overwritten to a fixed one, twice to the same value for a given bridge name.

An other side effect is that when interface is set administratively UP, the bridge operational status becomes DOWN instead of UNKNOWN initially because of this (see these answers of mine on SU and SF mentioning behaviors about DOWN and UNKNOWN: How does Linux determine the default MAC address of a bridge device? , linux ipv6 bridge address does not work when mac address is forced). Anyway this doesn't matter anymore once its first bridge port is attached.

Doing the same experiment inside a network namespace (eg: ip add netns experiment and ip netns exec experiment bash -l before running above commands twice) where systemd-udevd does not interfere will show the usual behavior of having different random addresses each time.

This is an effect of systemd ecosystem and doesn't happen on systems not running systemd (or older versions of systemd). One proposed fix is to use:

# /etc/systemd/network/90-bridge.link
[Match]
OriginalName=br*

[Link]
MACAddressPolicy=random

but it appears the real fix is to change the file that participates in generating this "stable random" value, as described there: https://wiki.debian.org/MachineId

Each machine should have a different value. This is especially important for cloned VMs from a base template. The relation between machine-id and the way the bridge "stable" MAC address is generated is mentioned in the patch having implemented the (quite breaking) change:

=== This patch

This patch means that we will set a "stable" MAC for pretty much any virtual device by default, where "stable" means keyed off the machine-id and interface name.

It was also mentioned that this would be having impacts , but this was shrugged off.

This is not limited to interfaces of type bridge but to any interface that would generate a random MAC address: for example types veth, macvlan tuntap are also affected.

I could verify that the same bridge name would get a different "stable random" value after doing the operations described in Debian's link:

rm -f /etc/machine-id /var/lib/dbus/machine-id
dbus-uuidgen --ensure=/etc/machine-id
dbus-uuidgen --ensure

giving now in previous ip monitor a new MAC address for the same bridge name: 32:ee:c8:92:9f:e8 instead of 1a:d0:fc:63:c1:71 when deleting and recreating brtest0.

Deleted 23: brtest0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default 
    link/ether 1a:d0:fc:63:c1:71 brd ff:ff:ff:ff:ff:ff
24: brtest0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default 
    link/ether da:72:b6:63:23:e5 brd ff:ff:ff:ff:ff:ff
24: brtest0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default 
    link/ether 32:ee:c8:92:9f:e8 brd ff:ff:ff:ff:ff:ff

Conclusion:

Because the bridge MAC address is now manually set the bridge won't inherit anymore one of the MAC addresses of other interfaces set as bridge ports, including the usual permanent (physical or VM's) interfaces which are expected to have each a different MAC address. Two systems using the same machine-id and the same bridge name (eg: br0) with such bridge participating in routing (ie: there's an IP address configured on the bridge, but even if not the bridge can emit other frames related to bridging depending on its settings) on the same LAN will emit frames with the same source MAC address (bridge's), possibly disrupting switches in the path and anyway ignoring such same source MAC address from the peer.

4
  • This Q/A explains the behavior seen in this SU Q/A: superuser.com/questions/1744190/…
    – A.B
    Oct 1 at 12:00
  • Thank you very much for that in-depth answer, accepted and +1. I have followed the procedure in the Debian link (the 3 lines of code in your answer); that solved the problem.
    – Binarus
    Oct 1 at 15:06
  • 3
    It’s worth noting that this aspect of misbehaving in the face of duplicate machine IDs is not unique to systemd. That file is expected to be unique to a system, and numerous other things can and do misbehave if it’s not. Oct 1 at 21:41
  • @AustinHemmelgarn Yes, that's true - +1. That's the reason why I prefer A.B's solution (and have written that in my own answer).
    – Binarus
    Oct 2 at 8:22
4

You can tell Debian to clone the MAC address with a bridge_hw directive.

e.g. my /etc/network/interfaces file:

iface enp2s0 inet manual

auto br0
iface br0 inet dhcp
  bridge_ports enp2s0
  bridge_hw enp2s0

Now both enp2s0 and br0 have the same address.

$ sudo dmesg | grep 18:d6
[    2.660733] r8169 0000:02:00.0 eth0: RTL8168e/8111e, 18:d6:c7:05:89:07, XID 2c2, IRQ 26

$ ifconfig enp2s0 | grep ether ; ifconfig br0 | grep Description:   Debian GNU/Linux 11 (bullseye)
        ether 18:d6:c7:05:89:07  txqueuelen 1000  (Ethernet)
        ether 18:d6:c7:05:89:07  txqueuelen 1000  (Ethernet)
1
  • Thank you very much, and +1. I already had seen bridge_hw and wondered what differentiates it from hwaddress ether. I didn't know yet that you can give enp2s0 as argument to bridge_hw instead of a literal MAC address. Very interesting - I have taken a note and will keep it in mind. However, A.B's answer also solves my problem, but in addition the problem of identical machine IDs on different machines, which likely would cause a lot of further issues.
    – Binarus
    Oct 2 at 8:28
2

First of all, A.B's answer shows the correct solution. If you are having the same problem, follow A.B's answer to solve it the right way.

However, for the sake of completeness, I'd like to show an alternative (much inferior) solution. I was under pressure and had not expected a correct answer so fast. Therefore I had temporarily solved the problem on my own by slightly changing /etc/network/interfaces:

auto br0
iface br0 inet static
        bridge_ports eno1
        address 192.168.20.11
        netmask 255.255.255.0
        broadcast 192.168.20.255
        gateway 192.168.20.250    
        hwaddress ether 02:01:01:01:11:01

The last line is the key point. It makes the OS assign the indicated MAC address to the bridge. The MAC address shown is an address from the semi-official private MAC address range. I have verified that this MAC address remains the same when you remove or add devices to the bridge.

Of course, you must give each bridge a different MAC address, regardless of whether or not the bridges are on the same machine.

Again, the above method only solves the bridge MAC address problem. I believe that a lot more can go heavily wrong if there are the same machine IDs on different machines. A.B's answer mentions how to create a new machine ID, and a quick test has shown that every machine ID created that way differs from every other machine ID created that way.

2
  • If you want the old behaviour (of inheriting the MAC address of the port) back, you should use bridge_hw eno1 instead of hwaddress. bridge_hw ca:fe:ca:fe:ca:fe is also recommended for setting a fixed/custom MAC address when you are using bridge-utils-interfaces. See manpages.debian.org/bullseye/bridge-utils/… and sources.debian.org/src/bridge-utils/1.7-1/debian/NEWS.
    – Bob
    Oct 1 at 19:43
  • @Bob Thanks for the tips, and +1. I'll keep that in mind. However, I now have deployed A.B's solution since having the same machine ID on different machines would be a bad thing anyway.
    – Binarus
    Oct 2 at 8:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.