0

We had email from Hetzner network team asking us not to use MAC addresses belonging to the Virtual Machines belonging to the subnet.

We configured Xen server host as a router using this guide.

After asking for additional details Hetzner support answered that normally, the network configuration of your hypervisor should only let packets exit the system with the MAC address of the real NIC. But if you do not find the issue, you might try to block those outgoing packets using IPtables.

So our questions:

If anyone had this kind of problem with Hetzner or other dedicated server provider.
How you solved it?

Thank you

Action plans that I tried and didn't help

  1. Create separate router VM
  2. Turn off IPv6 on all VMs and host using sysctl.conf directives
  3. Masquarading on both host interfaces

    -A POSTROUTING -o xenbr0 -j MASQUERADE
    -A POSTROUTING -o eth0 -j MASQUERADE 
    
  4. Bring up additional xenbr0:1 interface

  5. Adding following into /etc/sysctl.conf

    net.ipv4.conf.default.proxy_arp = 1
    

Configuration details

Host/Router configuration:

[root@xenserver-custom ~]# cat /etc/sysctl.conf

net.ipv4.ip_forward = 1  
net.ipv6.conf.all.forwarding=1

net.ipv4.conf.default.proxy_arp = 0  

net.ipv4.conf.all.send_redirects = 0  
net.ipv4.conf.default.send_redirects = 0  
net.ipv4.conf.lo.send_redirects = 0  
net.ipv4.conf.xenbr0.send_redirects = 0  

[root@xenserver-custom network-scripts]# ip addr add 85.91.107.177/28 dev xenbr0

[root@xenserver-custom ~]# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether 0c:c4:7a:e7:dc:33  txqueuelen 1000  (Ethernet)
        RX packets 4704816217  bytes 6002063739181 (5.4 TiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 6294828922  bytes 7643975899027 (6.9 TiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        loop  txqueuelen 1  (Local Loopback)
        RX packets 518545683  bytes 6322784653872 (5.7 TiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 518545683  bytes 6322784653872 (5.7 TiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vifxxxx

.....................

xenbr0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 115.35.61.184  netmask 255.255.255.192  broadcast 115.35.61.191
        ether 0c:c4:7a:e7:dc:33  txqueuelen 1  (Ethernet)
        RX packets 3070611738  bytes 8670969429554 (7.8 TiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 2680055664  bytes 9822630727363 (8.9 TiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

VM Guest configuration

[root@r1213a network-scripts]# cat /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
TYPE=Ethernet
ONBOOT=yes
NM_CONTROLLED=yes
BOOTPROTO=static
IPADDR=85.91.107.184
PREFIX=28
GATEWAY=85.91.107.177
DNS1=213.133.98.98
DEFROUTE=yes
IPV4_FAILURE_FATAL=yes
IPV6INIT=no

[root@r1213a network-scripts]# ifconfig
eth0      Link encap:Ethernet  HWaddr B6:8F:14:74:A6:B6
          inet addr:85.91.107.184  Bcast:85.91.107.191  Mask:255.255.255.240
          inet6 addr: fe80::b48f:14ff:fe74:a6b6/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:27122939 errors:0 dropped:2 overruns:0 frame:0
          TX packets:2218911 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:5404322465 (5.0 GiB)  TX bytes:1061055301 (1011.9 MiB)


[root@r1213a network-scripts]# ip route
default via 85.91.107.177 dev eth0
85.91.107.176/28 dev eth0  proto kernel  scope link  src 85.91.107.184

[root@r1213a network-scripts]# traceroute google.com
traceroute to google.com (172.217.18.110), 30 hops max, 60 byte packets
 1  xenserver.localdomain (85.91.107.177)  0.081 ms  0.029 ms  0.039 ms
 2  static.129.61.69.159.clients.your-server.de (159.69.61.129)  0.390 ms  0.410 ms  0.370 ms
 3  core22.fsn1.hetzner.com (213.239.245.121)  0.393 ms  0.416 ms  0.424 ms
 4  core0.fra.hetzner.com (213.239.252.33)  5.207 ms  5.184 ms core0.fra.hetzner.com (213.239.252.29)  5.049 ms
 5  72.14.218.94 (72.14.218.94)  5.273 ms  5.249 ms 72.14.218.176 (72.14.218.176)  4.990 ms
 6  108.170.251.193 (108.170.251.193)  5.139 ms *  5.019 ms
 7  209.85.241.75 (209.85.241.75)  5.834 ms 216.239.40.58 (216.239.40.58)  5.092 ms 172.253.64.119 (172.253.64.119)  5.707 ms
 8  108.170.251.144 (108.170.251.144)  15.292 ms zrh04s05-in-f110.1e100.net (172.217.18.110)  4.952 ms  4.903 ms
  • We cannot guess how the network/routing is implemented, please document it instead of pointing to guides/external links. We also do not know hetznet, please provide context. – Rui F Ribeiro Apr 2 at 5:28
  • Context about what kind of service. – Rui F Ribeiro Apr 2 at 5:35
  • Ok, to be more abstract - is there any possibility that after Linux host configured as a router outgoing packets still have MAC addresses unchanged? Our dedicated server provider claims so and asks us to fix it. – Alex Apr 2 at 5:43
  • the abstract answer is "yes". – Ralph Rönnquist Apr 2 at 5:46
  • I also added to the post configuration that we performed using sysctl.conf to setup host as a router. Is there any setting to be added in this or other confs to prevent packets with local network MAC addresses to go outside? – Alex Apr 2 at 5:49
1

I think this might be resolved if you make your host MASQUERADE outgoing packets, which you would do by an iptables rule like

iptables -t nat -A POSTROUTING -i eth0 -j MASQUERADE

That rule tells the kernel to rewrite outgoing packets on eth0 to be as of that interface, while it also keeps a back-mapping so it is able to return response traffic to originator.

It might however be the case that, since the VM packets are delivered through bridging rather than forwarding, the iptables rule does not take effect for that traffic. This link is a brisk summary of it.

  • [root@xenserver-enginatics1 network-scripts]# iptables -t nat -A POSTROUTING -i eth0 -j MASQUERADE iptables v1.4.21: Can't use -i with POSTROUTING Try iptables -h' or 'iptables --help' for more information. Should I use -o instead of -i? – Alex Apr 4 at 10:07
  • Also tried it on xenbr0 instead of eth0. Seems like on eth0 it didn't work, because guest IP still showed instead of NAT gateway. When I applied POSTROUTING rule to xenbr0 interface, it started to show gateway IP address as public. Here's command I used to verify from Linux guest VM: [root@r1213a ~]# curl ifconfig.me 85.91.107.184. – Alex Apr 4 at 10:10
  • Command didn't help if applied to either eth0 or xenbr0, I used -o option – Alex Apr 4 at 12:33
0

I applied solution from superuser.com. It worked for me

All credits to Alexey Degtyarev

Thanks to Hetzner network team also as they pointed to the solution, but I went to wrong direction trying to create second xenbr in the same network and assign VIFs to it. But It's not working because after virtual machine restart VIF number is regenerated.

In Hetzner you have two types of additional IP-addresses which you can use with a dedicated server: a single IPv4 & an IPv4 subnet. For each single IP you get a MAC-address and you must use that MAC on the network interface of a new VM instance. For each additional subnet you must setup a new network and setup routing between that network and the default network (associated with eth0) of the server.

In XenServer this can be done using Linux console:

xe network-create name-label="Additional network" name-description="46.xx.yy.zz/28"

Tip1: xapi0 bridge didn't show up on host until I assigned one of VMs to new network in Xen Center. Just tip that may save time if it happens to anyone else.

This will create a new network attached to a new bridge (xapi0 by default) in XenServer. Then assign the first usable IP-address of the network (according to it's netmask) to that bridge:

ip addr add 46.xx.yy.1/28 dev xapi0

Now you can add new VMs with autogenerated MACs attached to the newly created network instead of the default network. The traffic will be switched & routed inside XenServer.

Having that setup I got confirmation from Hetzner network team that only allowed MACs are visible on the switchport.

Tip2: You can verify there is no offending traffic on eth0 anymore from your side at host:

tcpdump -i eth0 -en|egrep -i 'mac1|mac2'

It all should be seen on xapi0 instead

tcpdump -i xapi0 -en|egrep -i 'mac1|mac2'

Where mac1,mac2 - MAC addresses reported by Hetzner as prohibited.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.