3

I have a physical server running Ubuntu Trusty (14.04) at a data centre where they assigned one IP address, as well as a /28 block (16 IP addresses, 14 usable) from a different subnet. They said that the additional subnet was statically routed via the physical host, which I assume means their network will only accept requests from the host NIC's hardware address.

I'm going to use this with virtualbox, so that each VM has their own IP address and works as if they were real physical servers on the net (in other words, no NAT). I further assume that the VMs need to route their traffic via the host in order for their requests to use the host's HW address.

I am not an expert on setting up complex networks beyond the basic static IP or NAT, so I've read man pages, Ubuntu documentation online as well as a bunch of questions and answers on different StackExchange sites. From what I gather, I need to set up a virtual bridge between the host IP address and one address in the subnet, the latter acting as the gateway for the virtual machines. The VMs will thus have a simple setup with the gateway being in their subnet.

I can access the VM from the host and vice versa, but the VMs can't access the internet beyond the host and I can't access the VMs from the internet. I'm assuming the problem is that I haven't set up the bridge properly to route the requests over it. Any hints to get me moving in the right direction would be highly appreciated.

Host's /etc/network/interfaces

auto lo
iface lo inet loopback

auto  br0
iface br0 inet static
  name      Bridge for VMs to connect via
  address   x.x.x.229
  netmask   255.255.255.192
  gateway   x.x.x.193
  # default route to access subnet - this was provided by the data centre:
  up route add -net x.x.x.192 netmask 255.255.255.192 gw x.x.x.193 eth0
  bridge_ports eth0
  bridge_fd 9
  bridge_hello 2
  bridge_maxage 12
  bridge_stp off

auto  br0:0
iface br0:0 inet static
  name      Gateway IP for VMs in subnet y.y.y.240/28
  address   y.y.y.241
  netmask   255.255.255.240
  broadcast y.y.y.255
  network   y.y.y.240

Host system info:

$ route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         x.x.x.193  0.0.0.0         UG    0      0        0 br0
x.x.x.192  0.0.0.0         255.255.255.192 U     0      0        0 br0
y.y.y.240  0.0.0.0         255.255.255.240 U     0      0        0 br0

$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br0 state UP group default qlen 1000
    link/ether c8:60:00:5e:bd:e0 brd ff:ff:ff:ff:ff:ff
3: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether c8:60:00:5e:bd:e0 brd ff:ff:ff:ff:ff:ff
    inet x.x.x.229/26 brd x.x.x.255 scope global br0
       valid_lft forever preferred_lft forever
    inet y.y.y.241/28 brd y.y.y.255 scope global br0:0
       valid_lft forever preferred_lft forever

$ ifconfig
br0       Link encap:Ethernet  HWaddr c8:60:00:5e:bd:e0  
          inet addr:x.x.x.229  Bcast:x.x.x.255  Mask:255.255.255.192
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:743907 errors:0 dropped:0 overruns:0 frame:0
          TX packets:519787 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:92549304 (92.5 MB)  TX bytes:172422977 (172.4 MB)

br0:0     Link encap:Ethernet  HWaddr c8:60:00:5e:bd:e0  
          inet addr:y.y.y.241  Bcast:y.y.y.255  Mask:255.255.255.240
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

eth0      Link encap:Ethernet  HWaddr c8:60:00:5e:bd:e0  
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:743968 errors:0 dropped:0 overruns:0 frame:0
          TX packets:519787 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:102968103 (102.9 MB)  TX bytes:172422977 (172.4 MB)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:674890 errors:0 dropped:0 overruns:0 frame:0
          TX packets:674890 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:302219962 (302.2 MB)  TX bytes:302219962 (302.2 MB)

Virtual Machine's /etc/network/interfaces

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
  address y.y.y.243
  netmask 255.255.255.240
  gateway y.y.y.241

Virtual Machine system info:

$ route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         y.y.y.241  0.0.0.0         UG    0      0        0 eth0
y.y.y.240  0.0.0.0         255.255.255.240 U     0      0        0 eth0

$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 08:00:27:f7:36:74 brd ff:ff:ff:ff:ff:ff
    inet y.y.y.243/28 brd y.y.y.255 scope global eth0
       valid_lft forever preferred_lft forever

$ ifconfig
eth0      Link encap:Ethernet  HWaddr 08:00:27:f7:36:74  
          inet addr:y.y.y.243  Bcast:y.y.y.255  Mask:255.255.255.240
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:18112 errors:0 dropped:0 overruns:0 frame:0
          TX packets:241 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:1123962 (1.1 MB)  TX bytes:54822 (54.8 KB)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:444 errors:0 dropped:0 overruns:0 frame:0
          TX packets:444 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:35428 (35.4 KB)  TX bytes:35428 (35.4 KB)

$ ping -c 4 -W 5 x.x.x.229
PING x.x.x.229 (x.x.x.229) 56(84) bytes of data.
64 bytes from x.x.x.229: icmp_seq=1 ttl=64 time=0.117 ms
64 bytes from x.x.x.229: icmp_seq=2 ttl=64 time=0.106 ms
64 bytes from x.x.x.229: icmp_seq=3 ttl=64 time=0.102 ms
64 bytes from x.x.x.229: icmp_seq=4 ttl=64 time=0.103 ms

--- x.x.x.229 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3000ms
rtt min/avg/max/mdev = 0.102/0.107/0.117/0.006 ms

$ ping -c 4 -W 5 x.x.x.193
PING x.x.x.193 (x.x.x.193) 56(84) bytes of data.

--- x.x.x.193 ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 3007ms

$ ping -c 4 -W 5 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.

--- 8.8.8.8 ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 3023ms

In Virtualbox, I've set up the VM's virtual network interface as a Bridged Adapter connected to br0. I also tried connecting it to eth0, but it made no discernible difference.

UPDATE:

Following advice in the comment, I've enabled IP forwarding:

$ sysctl net.ipv4.ip_forward
net.ipv4.ip_forward = 0
$ sudo sysctl -w net.ipv4.ip_forward=1
net.ipv4.ip_forward = 1

I've also enabled promiscuous mode in virtualbox network settings, although I don't think this made any difference.

I've noticed something interesting. When I ping something on the net, such as Google's name servers, I get a reply on the first ping, but not on the subsequent ones:

$ ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
From y.y.y.241: icmp_seq=1 Redirect Host(New nexthop: y.y.y.y.241)
64 bytes from y.y.y.241: icmp_seq=1 ttl=54 time=5.97 ms
From y.y.y.241 icmp_seq=2 Destination Host Unreachable
From y.y.y.241 icmp_seq=3 Destination Host Unreachable
From y.y.y.241 icmp_seq=4 Destination Host Unreachable
From y.y.y.241 icmp_seq=5 Destination Host Unreachable
From y.y.y.241 icmp_seq=6 Destination Host Unreachable
From y.y.y.241 icmp_seq=7 Destination Host Unreachable
^C
--- 8.8.8.8 ping statistics ---
9 packets transmitted, 1 received, +6 errors, 88% packet loss, time 8041ms
rtt min/avg/max/mdev = 5.972/5.972/5.972/0.000 ms, pipe 3

When IP forwarding is off, all I get is:

$ ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
^C
--- 8.8.8.8 ping statistics ---
11 packets transmitted, 0 received, 100% packet loss, time 10000ms

This is what I get if I sniff the IP packets on the host when pinging 8.8.4.4 on the VM for the first time since rebooting the VM. Subsequent pings to the same IP address only gives the repeated lines on the bottom (not the first two).

$ sudo tcpdump -q -c 10 -e -n host y.y.y.243
tcpdump: WARNING: eth0: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
12:32:57.199202 c8:60:00:5e:bd:e0 > cc:e1:7f:07:e0:af, IPv4, length 98: y.y.y.243 > 8.8.4.4: ICMP echo request, id 1060, seq 1, length 64
12:32:57.204498 cc:e1:7f:07:e0:af > c8:60:00:5e:bd:e0, IPv4, length 98: 8.8.4.4 > y.y.y.243: ICMP echo reply, id 1060, seq 1, length 64
12:32:58.202555 08:00:27:f7:36:74 > ff:ff:ff:ff:ff:ff, ARP, length 60: Request who-has x.x.x.193 tell y.y.y.243, length 46
12:32:59.201628 08:00:27:f7:36:74 > ff:ff:ff:ff:ff:ff, ARP, length 60: Request who-has x.x.x.193 tell y.y.y.243, length 46
12:33:00.201652 08:00:27:f7:36:74 > ff:ff:ff:ff:ff:ff, ARP, length 60: Request who-has x.x.x.193 tell y.y.y.243, length 46
12:33:01.201759 08:00:27:f7:36:74 > ff:ff:ff:ff:ff:ff, ARP, length 60: Request who-has x.x.x.193 tell y.y.y.243, length 46
12:33:02.201502 08:00:27:f7:36:74 > ff:ff:ff:ff:ff:ff, ARP, length 60: Request who-has x.x.x.193 tell y.y.y.243, length 46
12:33:03.201394 08:00:27:f7:36:74 > ff:ff:ff:ff:ff:ff, ARP, length 60: Request who-has x.x.x.193 tell y.y.y.243, length 46
12:33:04.201706 08:00:27:f7:36:74 > ff:ff:ff:ff:ff:ff, ARP, length 60: Request who-has x.x.x.193 tell y.y.y.243, length 46
12:33:05.201647 08:00:27:f7:36:74 > ff:ff:ff:ff:ff:ff, ARP, length 60: Request who-has x.x.x.193 tell y.y.y.243, length 46
10 packets captured
11 packets received by filter
0 packets dropped by kernel

What could possibly cause such an odd result? What else could I test that could get me closer to a solution here?

UPDATE 2:

I am officially baffled over this one now. I ssh'd the host's IP address within the additional subnet (y.y.y.241) from the net, just to make sure that routing to that subnet works, and it did. I then ssh'd y.y.y.243 and expected to get nothing, but to my surprise I got in! Suddenly everything worked as it should... I could ping 8.8.8.8 and anything else and ssh to any internet host. And this is after doing nothing to either host or VM.

So of course I did the right thing - I broke it in the name of science. I had to find out what made it work! So, on the host:

sudo sysctl -w net.ipv4.ip_forward=0

Obviously I enabled it again:

sudo sysctl -w net.ipv4.ip_forward=1

And as expected, back to the situation above. Can't contact the VM other than from the host. On the VM, ping to any internet host has a first reply, then nothing.

Now the really puzzling thing is, I then just left it for a few hours, and when I came back - and I stress that I did nothing to either host or VM - it magically worked. I do not appreciate magic. I'm a man of science and I like things to either work, or not to work, preferably the former. Do anyone have any idea what could cause such a strange delay?

  • Is IP Forwarding enabled on the host? – garethTheRed Oct 10 '15 at 19:52
  • I don't know. How would I know whether it is enabled or not, and how is it enabled if it happens to be disabled? – Ronny Ager-Wick Oct 10 '15 at 20:07
  • sysctl net.ipv4.ip_forward should return 1. If it returns 0 then: sudo sysctl -w net.ipv4.ip_forward=1 will set it. If that works, edit /etc/sysctl.conf to add it permanently (it may already be there, but commented out). – garethTheRed Oct 11 '15 at 6:54
  • @garethTheRed Thank you, this seems to get me one step closer. I've updated the post to reflect your suggestion. Still doesn't work, but it definitely made a difference! – Ronny Ager-Wick Oct 12 '15 at 9:45
  • Destination host unreachable means that the routing to the subnet is working, but the host in not reachable from that subnet. This message is generated by the router listed in the message (y.y.y.241). So basically y.y.y.241 can't reach 8.8.8.8. That's because you've disabled ip_forward. Re-enable it and it should work. – garethTheRed Oct 12 '15 at 13:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.