5

I'm trying to setup an RHCE study lab per Michael Jang's RHCSA/RHCE setup guide. I've followed the guide in the book to the dot, though in my opinion a lot of information is missing (step by step would have been nice for such a book).

TL;DR - I cannot get VMs to talk to each other in the same subnet in my KVM setup. My study guide says to use Forward NAT with the virbr# devices...but it is not working.

The main issue I am having is that my VMs cannot ping anything on the networks: "destination host unreachable". server1 cannot ping its default gateway, the host computer, or even tester1, which is on the same subnet. The same is true for trying to reach outsider1, which is a different subnet on the same KVM host PC. The same behavior is present on the VMs tester1 and outsider1 too.

For the setup, I have a host computer on a private network (192.168.5.0/24), and per the book I created two KVM virtual networks and three VMs. Below is a summary of the configuration for each VM.

server1.example.com

  • attached to virtual network 'example.com': NAT
  • device model: virtio
  • vNIC MAC 52:54:00:86:51:d2
  • static IP: 192.168.122.50/24, gw=192.168.122.1

tester1.example.com

  • attached to virtual network 'example.com': NAT
  • device model: virtio
  • vNIC MAC 52:54:00:89:20:c7
  • static IP: 192.168.122.150/24, gw=192.168.122.1

outsider1.example.org

  • attached to virtual network 'example.org': NAT
  • device model: virtio
  • vNIC MAC 52:54:00:03:c3:0a
  • static IP: 192.168.100.100/24, gw=192.168.100.1

I followed the directions to create two virtual networks, and the setup as seen from the virtualization host PC is as follows:

# virsh list
 Id    Name                           State
----------------------------------------------------
 1     outsider1                      running
 2     tester1                        running
 4     server1                        running

# virsh net-list
 Name                 State      Autostart     Persistent
----------------------------------------------------------
 example.com          active     yes           yes
 example.org          active     yes           yes

# virsh net-info example.com
Name:           example.com
UUID:           6d2a6e12-2d72-4720-9427-630a608bae6f
Active:         yes
Persistent:     yes
Autostart:      yes
Bridge:         virbr0

# virsh net-info example.org
Name:           example.org
UUID:           3d564af8-4d3e-484b-846e-7ad76bd4be4a
Active:         yes
Persistent:     yes
Autostart:      yes
Bridge:         virbr1

# virsh net-dumpxml example.com
<network>
  <name>example.com</name>
  <uuid>6d2a6e12-2d72-4720-9427-630a608bae6f</uuid>
  <forward mode='nat'>
    <nat>
      <port start='1024' end='65535'/>
    </nat>
  </forward>
  <bridge name='virbr0' stp='on' delay='0'/>
  <mac address='52:54:00:7f:b9:50'/>
  <domain name='example.com'/>
  <ip address='192.168.122.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='192.168.122.151' end='192.168.122.254'/>
    </dhcp>
  </ip>
  <ip family='ipv6' address='fd00:a81d:a6d7:55::1' prefix='64'>
    <dhcp>
      <range start='fd00:a81d:a6d7:55::100' end='fd00:a81d:a6d7:55::1ff'/>
    </dhcp>
  </ip>
</network>

# virsh net-dumpxml example.org
<network>
  <name>example.org</name>
  <uuid>3d564af8-4d3e-484b-846e-7ad76bd4be4a</uuid>
  <forward mode='nat'>
    <nat>
      <port start='1024' end='65535'/>
    </nat>
  </forward>
  <bridge name='virbr1' stp='on' delay='0'/>
  <mac address='52:54:00:49:c7:35'/>
  <domain name='example.org'/>
  <ip address='192.168.100.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='192.168.100.128' end='192.168.100.254'/>
    </dhcp>
  </ip>
  <ip family='ipv6' address='fd00:e81d:a6d7:56::1' prefix='64'>
    <dhcp>
      <range start='fd00:e81d:a6d7:56::100' end='fd00:e81d:a6d7:56::1ff'/>
    </dhcp>
  </ip>
</network>

# brctl show
bridge name bridge id       STP enabled interfaces
docker0     8000.02428a3f4914   no      
virbr0      8000.5254007fb950   yes     virbr0-nic
virbr1      8000.52540049c735   yes     virbr1-nic

# ip route show
default via 192.168.5.1 dev enp0s31f6 proto dhcp metric 100 
192.168.5.0/24 dev enp0s31f6 proto kernel scope link src 192.168.5.45 metric 100 
192.168.100.0/24 dev virbr1 proto kernel scope link src 192.168.100.1 
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 

# ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp0s31f6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 2c:4d:54:d2:c5:89 brd ff:ff:ff:ff:ff:ff
    inet 192.168.5.45/24 brd 192.168.5.255 scope global noprefixroute dynamic enp0s31f6
       valid_lft 71762sec preferred_lft 71762sec
    inet6 fe80::7abc:be60:6633:d94/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:8a:3f:49:14 brd ff:ff:ff:ff:ff:ff
10: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 1000
    link/ether fe:54:00:03:c3:0a brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc54:ff:fe03:c30a/64 scope link 
       valid_lft forever preferred_lft forever
11: vnet1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 1000
    link/ether fe:54:00:89:20:c7 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc54:ff:fe89:20c7/64 scope link 
       valid_lft forever preferred_lft forever
13: vnet2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 1000
    link/ether fe:54:00:86:51:d2 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc54:ff:fe86:51d2/64 scope link 
       valid_lft forever preferred_lft forever
14: virbr1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 52:54:00:49:c7:35 brd ff:ff:ff:ff:ff:ff
    inet 192.168.100.1/24 brd 192.168.100.255 scope global virbr1
       valid_lft forever preferred_lft forever
    inet6 fd00:e81d:a6d7:56::1/64 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fe49:c735/64 scope link 
       valid_lft forever preferred_lft forever
15: virbr1-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr1 state DOWN group default qlen 1000
    link/ether 52:54:00:49:c7:35 brd ff:ff:ff:ff:ff:ff
16: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 52:54:00:7f:b9:50 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
    inet6 fd00:a81d:a6d7:55::1/64 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fe7f:b950/64 scope link 
       valid_lft forever preferred_lft forever
17: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN group default qlen 1000
    link/ether 52:54:00:7f:b9:50 brd ff:ff:ff:ff:ff:ff

I have tried restart NetworkManager on all of the VMs, as well as restarting the VMs too, but none of the VMs can talk to any other device. Clearly I am missing a step in order to get these devices to talk to each other...

Server1 is configured as follows:

[root@server1 ~]# ip route show
default via 192.168.122.1 dev eth0  proto static  metric 100 
192.168.122.0/24 dev eth0  proto kernel  scope link  src 192.168.122.50  metric 100 
[root@server1 ~]# ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT qlen 1000
    link/ether 52:54:00:86:51:d2 brd ff:ff:ff:ff:ff:ff
[root@server1 ~]# ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:86:51:d2 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.50/24 brd 192.168.122.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fe86:51d2/64 scope link 
       valid_lft forever preferred_lft forever

Tester1 is configured as follows:

[root@tester1 ~]# ip route show
default via 192.168.122.1 dev eth0  proto static  metric 100 
192.168.122.0/24 dev eth0  proto kernel  scope link  src 192.168.122.150  metric 100 
[root@tester1 ~]# ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT qlen 1000
    link/ether 52:54:00:89:20:c7 brd ff:ff:ff:ff:ff:ff
[root@tester1 ~]# ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:89:20:c7 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.150/24 brd 192.168.122.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fe89:20c7/64 scope link 
       valid_lft forever preferred_lft forever

And outsider1 is configured as follows:

[root@outsider1 ~]# ip route show
default via 192.168.100.1 dev eth0  proto static  metric 100 
192.168.100.0/24 dev eth0  proto kernel  scope link  src 192.168.100.100  metric 100 
[root@outsider1 ~]# ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT qlen 1000
    link/ether 52:54:00:03:c3:0a brd ff:ff:ff:ff:ff:ff
[root@outsider1 ~]# ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:03:c3:0a brd ff:ff:ff:ff:ff:ff
    inet 192.168.100.100/24 brd 192.168.100.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fe03:c30a/64 scope link 
       valid_lft forever preferred_lft forever

I'm pretty sure I will need to create static routes on the hosts to get traffic going from outsider1 to tester1/server1 and vice versa. But what really has me stuck is that I cannot even get tester1 and server1 to talk to each other, or even talk to the default gateway virbr0.

Any ideas?

  • If you restart a virtual network, VMs on that network may lose connectivity until you shut them down and then start them again. – Michael Hampton May 5 at 20:52
  • Thanks for the tip. I've tried restarting NetworkManager as well as "systemctl reboot" a few times on the VMs, but no luck on the connectivity. – Dr. Watson May 6 at 16:43
  • Reboot is not sufficient to restore connectivity in this scenario; you have to actually shutdown the VM and start it again. – Michael Hampton May 9 at 3:20
  • I know little about virsh but I can see your host's slave interfaces are down. Try bring them up manually. At the system level you do ip link set virbr0-nic up and the same for virbr1-nic, and you can probably use this system command as a test, but I trust there is an equivalent through virsh and you should rather use that – LL3 May 10 at 12:52
  • @MichaelHampton: I have tried it all, reboot, power down VM, power down host, etc. – Dr. Watson May 11 at 4:28
2
+75

That setup imply a lot of important parameter that are not shown on the question (iptables/used steps/kernel network config/br setup/etc) here are some possibilities to solve the communication issue (in the master host side):

Possible solution:

A bridge interface as its name indicate is a gateway between multiple network interfaces in your setup it seem that you have:

[KVM1-ETH0] <---> [Bridge][virbr0] <---> [Master][Bridge][virbr0-nic]
[KVM2-ETH0] <---> [Bridge][virbr1] <---> [Master][Bridge][virbr1-nic]

First i don't understand why you need 2 bridges interfaces and then the second problem as @LL3 mentioned the interface virbr0-nic is not turned on.

Second a more clean setup would look like this (depending on your needs)

[KVM1-ETH0] <---> [Bridge][virbr0] <---> [Master][Bridge][virbr0-nic]
[KVM2-ETH0] <---> [Bridge][virbr1] <---> [Master][Bridge][virbr0-nic]

Or

[KVM1-ETH0] <---> [Bridge][virbr0] <---> [Master][enp0s31f6]
[KVM2-ETH0] <---> [Bridge][virbr0] <---> [Master][enp0s31f6]

Or

[KVM1-ETH0] <---> [Bridge][virbr0]
[KVM2-ETH0] <---> [Bridge][virbr0]

Possible solution:

Bring up virbr0-nic and virbr1-nic (adapt the ip to your needs)

ifconfig virbr0-nic 192.168.122.254/24 up
ifconfig virbr1-nic 192.168.122.254/24 up

But still virbr0-nic and virbr1-nic are not linked together.

Possible solution:

If you want to use your main interface you can add it as master to your bridges (depending on your needs)

ip link set enp0s31f6 master virbr0 
ip link set enp0s31f6 master virbr1

Possible solution:

Check your iptables/firewall setup (firewall like shorewall for instance if you use that), for a testing purpose you could empty the iptables before starting your setup.

#Netfilter cleanup
iptables --flush
iptables -t nat -F
iptables -X
iptables -Z
iptables -P INPUT ACCEPT
iptables -P OUTPUT ACCEPT
iptables -P FORWARD ACCEPT

Possible solution:

Host routing/forward feature. Normally ip_forward kernel feature is not required for bridges interfaces to work but this could help if the bridge is not well configured in the first place. (note that enabling ip_forward make the the master host act as a router, in a production environment this require additional precaution with iptables or other)

#Enabling ipv4 forwarding
echo "1" > /proc/sys/net/ipv4/ip_forward

Possible solution:

Following the ip_forward possible solution you can use masquerade to force a bridge traffic to use a specific interface

iptables -t nat -A POSTROUTING -o eth0 -s 192.168.0.0/24  -j MASQUERADE 

In this example the traffic of the bridge with the ip 192.168.0.0/24 will be forwarded to eth0 (this require ip_forward)

Possible solution:

You could consider other setup solution (other than bridge interface) like macvlan, ipvlan or veth interfaces (some examples)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.