My initial state is something like this:
- A "normal" home network with a residential gateway that hands out DHCP addresses to a combination of wired and wireless machines on subnet 192.168.15.0/24.
- A linux (Ubuntu 18.04.3) virtual machine host with a mix of windows and linux guests. This VM host uses a single bridge interface to directly bridge the VM guest interfaces into the 192.168.15.x network. Here eno1 and ens2 are physical ethernet ports connecting the VM host to the main network and "extending" this bridge domain to another physical switch, and the vnetX interfaces are the interfaces of the VM guest machines.
lwobker@lwobker-vms:~$ brctl show bridge name bridge id STP enabled interfaces bridge1 8000.0017b60066e8 no eno1 # goes to router/gateway ens2 # extends the subnet to other machines vnet0 # VM guest #1 vnet1 # VM guest #2 docker0 8000.0242bd3d4632 no virbr0 8000.52540096aaf5 yes virbr0-nic
This all worked as I wanted it to, where local wired, wireless, and VM guest clients could all get DHCP addresses from the main 192.168.15.1 res-gateway/router, some of which were reserved/static on the DHCP server side and some were dynamic. Anyway - all good... UNTIL...
I installed Docker. I did this to run the Ubiquity/Unifi wireless AP controller software in a container, it seemed like a good/simple use case to play around with containers. However, somewhere along the way the installation of Docker (and I'm pretty sure more accurately the Docker networking changes) broke DHCP for anything BEHIND the VM host machines bridge interface. To be explicit:
- wired and wireless clients that do NOT sit behind the VM host machine still receive DHCP addresses normally from 192.168.15.1 (both reserved and dynamic)
- all the machines that sit behind the VM host bridge no longer receive DHCP addresses. This includes BOTH virtual machine guest instances on the VM itself, as well as machines that are physically cabled through interface ens2.
- importantly: IPv6 addressing does still work on machines behind this newly-Dockerfied VM host, and these machines can still get out to the internet successfully as long as the target is a routable v6 address.
I'm obviously new to docker and have a decent grasp of linux networking and bridging, but this is past my abilities. I've fired up tcpdump a couple of times and I can see the DHCP From reading around it may well be something related to iptables - I do not have the output of iptables --list from before the docker install, but there are obviously entries that were added ;-) I've included whatever I can convince myself is relevant below and tried to annotate entries that are particularly important. Help?!?!!
lwobker@lwobker-vms:~$ brctl showmacs bridge1 | uniq
port no mac addr is local? ageing timer
1 18:1d:ea:8a:86:e9 no 49.83
1 44:61:32:d0:43:02 no 117.27
1 44:61:32:fb:4d:7b no 117.27
1 60:6d:c7:1a:8f:e1 no 4.47
1 78:f2:9e:90:4c:a1 no 1.21
1 7c:2e:bd:9c:4a:4a no 0.00
1 80:4a:14:ec:5a:ea no 95.33
1 80:4a:14:f3:d8:8c no 71.62
1 90:70:65:13:b7:16 no 117.16
1 a0:cc:2b:ff:a4:b4 no 0.70
1 ac:1f:6b:b3:ad:fa yes 0.00
1 ac:1f:6b:b7:d2:44 no 286.56
1 b4:fb:e4:d6:e2:35 no 2.54
1 b8:27:eb:f9:c6:fe no 117.18
1 d8:31:34:f3:2c:69 no 62.38
1 ec:11:27:58:a8:0d no 0.22
2 00:17:b6:00:66:e8 yes 0.00
2 00:30:93:10:05:8e no 3.61 >> physical machine behind intf ens2
3 52:54:00:ed:c9:fc no 0.67 >> linux VM guest
3 fe:54:00:ed:c9:fc yes 0.00
4 52:54:00:d9:2d:b0 no 1.29 >> windows VM guest
4 fe:54:00:d9:2d:b0 yes 0.00
Current iptables output:
lwobker@lwobker-vms:/storage/unifi$ sudo iptables --list
Chain INPUT (policy ACCEPT)
target prot opt source destination
ACCEPT udp -- anywhere anywhere udp dpt:domain
ACCEPT tcp -- anywhere anywhere tcp dpt:domain
ACCEPT udp -- anywhere anywhere udp dpt:bootps
ACCEPT tcp -- anywhere anywhere tcp dpt:bootps
ACCEPT udp -- anywhere anywhere multiport dports mdns
ACCEPT tcp -- anywhere anywhere multiport dports 4000
Chain FORWARD (policy DROP)
target prot opt source destination
ACCEPT all -- anywhere 192.168.122.0/24 ctstate RELATED,ESTABLISHED
ACCEPT all -- 192.168.122.0/24 anywhere
ACCEPT all -- anywhere anywhere
REJECT all -- anywhere anywhere reject-with icmp-port-unreachable
REJECT all -- anywhere anywhere reject-with icmp-port-unreachable
DOCKER-ISOLATION-STAGE-1 all -- anywhere anywhere
ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
DOCKER all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
ACCEPT udp -- anywhere anywhere udp dpt:bootpc
Chain DOCKER (1 references)
target prot opt source destination
Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target prot opt source destination
DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere
RETURN all -- anywhere anywhere
Chain DOCKER-ISOLATION-STAGE-2 (1 references)
target prot opt source destination
DROP all -- anywhere anywhere
RETURN all -- anywhere anywhere
lwobker@lwobker-vms:/storage/unifi$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
be342a77a105 jacobalberty/unifi:stable "/usr/local/bin/dock…" 23 hours ago Up 5 hours (healthy) unifi
Output from docker network inspect for the bridge...
lwobker@lwobker-vms:/storage/unifi$ docker network inspect 20d74e3d7efc
[
{
"Name": "bridge",
"Id": "20d74e3d7efc41320c054dbd4fde76808a7c0e021e737e22cfceebefacb24b8c",
"Created": "2019-10-15T12:36:22.059240054-04:00",
"Scope": "local",
"Driver": "bridge",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "172.17.0.0/16",
"Gateway": "172.17.0.1"
}
]
},
"Internal": false,
"Attachable": false,
"Ingress": false,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
"Containers": {},
"Options": {
"com.docker.network.bridge.default_bridge": "true",
"com.docker.network.bridge.enable_icc": "true",
"com.docker.network.bridge.enable_ip_masquerade": "true",
"com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
"com.docker.network.bridge.name": "docker0",
"com.docker.network.driver.mtu": "1500"
},
"Labels": {}
}
]
the ifconfig output from the VM host...
lwobker@lwobker-vms:~$ ifconfig | egrep -v 'errors|0.0 B|device mem'
bridge1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.15.150 netmask 255.255.255.0 broadcast 192.168.15.255
ether 00:17:b6:00:66:e8 txqueuelen 1000 (Ethernet)
RX packets 105730 bytes 73076273 (73.0 MB)
TX packets 47591 bytes 11607902 (11.6 MB)
docker0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
inet 172.17.0.1 netmask 255.255.0.0 broadcast 172.17.255.255
ether 02:42:bd:3d:46:32 txqueuelen 0 (Ethernet)
eno1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether ac:1f:6b:b3:ad:fa txqueuelen 1000 (Ethernet)
RX packets 479110 bytes 451330614 (451.3 MB)
TX packets 352901 bytes 284906323 (284.9 MB)
eno2: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
ether ac:1f:6b:b3:ad:fb txqueuelen 1000 (Ethernet)
eno2:avahi: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
inet 169.254.11.170 netmask 255.255.0.0 broadcast 169.254.255.255
ether ac:1f:6b:b3:ad:fb txqueuelen 1000 (Ethernet)
ens2: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
ether 00:17:b6:00:66:e8 txqueuelen 1000 (Ethernet)
RX packets 375663 bytes 281196261 (281.1 MB)
TX packets 292663 bytes 270964344 (270.9 MB)
ens2:avahi: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
inet 169.254.8.164 netmask 255.255.0.0 broadcast 169.254.255.255
ether 00:17:b6:00:66:e8 txqueuelen 1000 (Ethernet)
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
loop txqueuelen 1000 (Local Loopback)
RX packets 92697 bytes 34802204 (34.8 MB)
TX packets 92697 bytes 34802204 (34.8 MB)
virbr0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
inet 192.168.122.1 netmask 255.255.255.0 broadcast 192.168.122.255
ether 52:54:00:96:aa:f5 txqueuelen 1000 (Ethernet)
virbr0-nic: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
ether 52:54:00:96:aa:f5 txqueuelen 1000 (Ethernet)
vnet0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fe:54:00:ed:c9:fc txqueuelen 1000 (Ethernet)
RX packets 4563 bytes 509949 (509.9 KB)
TX packets 28463 bytes 2796658 (2.7 MB)
vnet1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fe:54:00:d9:2d:b0 txqueuelen 1000 (Ethernet)
RX packets 50067 bytes 5253268 (5.2 MB)
TX packets 82339 bytes 105281318 (105.2 MB)
vnet0:avahi: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 169.254.12.44 netmask 255.255.0.0 broadcast 169.254.255.255
ether fe:54:00:ed:c9:fc txqueuelen 1000 (Ethernet)
vnet1:avahi: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 169.254.8.4 netmask 255.255.0.0 broadcast 169.254.255.255
ether fe:54:00:d9:2d:b0 txqueuelen 1000 (Ethernet)
sudo sysctl net.bridge.bridge-nf-call-iptablesreturnsnet.bridge.bridge-nf-call-iptables = 1? If not forget about this comment, if yes, you might have issues related to bridge plus iptables (yes the site's cert has expired), and it's hard to debug without all your configs (useiptables-save -cto dump all of iptables)