I've already asked this question on the #networking and #freebsd IRC channels on Freenode but nobody was able to answer my question and forwarded me over here as it seems this issue is kicky to solve.
I have a Proxmox hypervisor where are hosted LXC containers and KVM machines going from Debian 9, Arch Linux and Windows. All of them are bridged to the hypervisor and the IPv6 stack is working flawlessly with public IP addresses given by my ISP and routed directly to the VMs.
One of the VM I have is an OPNsense firewall appliance (based on FreeBSD 11.0-RELEASE-p8).
Contrary to the other LXC/KVM, the latter cannot ping any IPv6 machines outside those accessible directly from the bridge. As soon I try to ping either the gateway of my hypervisor (still in IPv6), or any other far away IPv6 hosts (e.g. google.com), I got the following error message:
ping6: sendmsg: No buffer space available
- At first, I thought the issue was due to issues with VirtIO drivers (bundled with FreeBSD). So I switched to emulated Intel E1000 NIC, but the problem persists. (I'm back with VirtIO now).
- A netstat -m reports the buffers as empty, so the problem doesn't come from here either.
- Putting the interface down and up again or rebooting doesn't fix the issue.
- I tested with a fresh FreeBSD and OpenBSD install (to avoid the OPNsense overlay), but the problem persists as well.
Pinging the VM either from the bridge or from a machine completely outside of the infrastructure doesn't respond, nor connect (i.e. I had started sshd on 2222, but weren't able to connect).
The FreeBSD host is configured like this:
ifconfig vtnet0 <ipv4>/32
route add <gw ipv4> -iface vtnet0
route add default <gw ipv4>
ifconfig vtnet0 inet6 <ipv6> prefixlen 64
route add -inet6 <gw ipv6> -iface vtnet0
route add -inet6 default <gw ipv6>
Please note all my GWs are outside of my IP subnets.
After applying these lines, the routes reported by netstat -rn are sensible to me. Nothing wrong.
I precise the pf firewall is completely disabled (pfctl -d). I want to make sure this is working flawlessly before enabling yet another level of failures. :)
Is there a bug somewhere in the BSD IPv6 stack as Linux is not complaining at all? This sounds weird as I think I'm not the only one in this situation and not the only one having GW outside their IP ranges.
Thanks in advance for your time / help.