0

I have a persistent problem with my virtual machine host. Host machine rebooted recently and after that one out of the two virtual networks stopped working. The network topology that I have is a /26 slice of B-class network at 145.120.88.64/26. Router is at address .126 and server is assigned address .65 Server machine is running on CentOS 7 linux. I have defined two virtual networks with /28 mask like so:

<network>
  <name>vmpublic</name>
  <uuid>065450c0-0429-48ba-9200-e4712cbe7283</uuid>
  <forward dev='em2' mode='route'>
    <interface dev='em2'/>
  </forward>
  <bridge name='virbr1' stp='on' delay='1'/>
  <mac address='52:54:00:e1:01:2a'/>
  <domain name='vmpublic'/>
  <ip address='145.120.88.81' netmask='255.255.255.240'>
    <dhcp>
      <range start='145.120.88.82' end='145.120.88.94'/>
      <host mac='52:54:00:67:03:83' name='vm1' ip='145.120.88.93'/>
    </dhcp>
  </ip>
</network>

Second network is defined like this:

<network>
  <name>vmpublicb</name>
  <uuid>05115dea-24a6-4b6d-9c39-38e852d71196</uuid>
  <forward dev='em2' mode='route'>
    <interface dev='em2'/>
  </forward>
  <bridge name='virbr2' stp='on' delay='0'/>
  <mac address='52:54:00:33:4a:81'/>
  <domain name='vmpublicb'/>
  <ip address='145.120.88.97' netmask='255.255.255.240'>
    <dhcp>
      <range start='145.120.88.98' end='145.120.88.110'/>
      <host mac='52:54:00:81:b5:0b' name='vm2' ip='145.120.88.100'/>
    </dhcp>
  </ip>
</network>

Then routing table is like this:

Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         145.120.88.126       0.0.0.0         UG    0      0        0 em2
145.120.88.64        0.0.0.0         255.255.255.192 U     0      0        0 em2
145.120.88.80        0.0.0.0         255.255.255.240 U     0      0        0 virbr1
145.120.88.96        0.0.0.0         255.255.255.240 U     0      0        0 virbr2
169.254.0.0     0.0.0.0         255.255.0.0     U     1003   0        0 em2
169.254.0.0     0.0.0.0         255.255.0.0     U     1006   0        0 virbr2
169.254.0.0     0.0.0.0         255.255.0.0     U     1009   0        0 virbr1

Firewall rules are like so:

Chain INPUT (policy ACCEPT 842 packets, 76357 bytes)
 pkts bytes target     prot opt in     out     source               destination         
15505 1222K ACCEPT     udp  --  virbr1 any     anywhere             anywhere             udp dpt:domain
    0     0 ACCEPT     tcp  --  virbr1 any     anywhere             anywhere             tcp dpt:domain
    6  1884 ACCEPT     udp  --  virbr1 any     anywhere             anywhere             udp dpt:bootps
    0     0 ACCEPT     tcp  --  virbr1 any     anywhere             anywhere             tcp dpt:bootps
 208K   16M ACCEPT     udp  --  virbr2 any     anywhere             anywhere             udp dpt:domain
    0     0 ACCEPT     tcp  --  virbr2 any     anywhere             anywhere             tcp dpt:domain
   18  5673 ACCEPT     udp  --  virbr2 any     anywhere             anywhere             udp dpt:bootps
    0     0 ACCEPT     tcp  --  virbr2 any     anywhere             anywhere             tcp dpt:bootps

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     all  --  em2    virbr1  anywhere             145.120.88.80/28   
 2508  191K ACCEPT     all  --  virbr1 em2     145.120.88.80/28          anywhere            
    0     0 ACCEPT     all  --  virbr1 virbr1  anywhere             anywhere            
    0     0 REJECT     all  --  any    virbr1  anywhere             anywhere             reject-with icmp-port-unreachable
    0     0 REJECT     all  --  virbr1 any     anywhere             anywhere             reject-with icmp-port-unreachable
14902 2099K ACCEPT     all  --  em2    virbr2  anywhere             145.120.88.96/28   
14782 2490K ACCEPT     all  --  virbr2 em2     145.120.88.96/28          anywhere            
    0     0 ACCEPT     all  --  virbr2 virbr2  anywhere             anywhere            
    0     0 REJECT     all  --  any    virbr2  anywhere             anywhere             reject-with icmp-port-unreachable
    0     0 REJECT     all  --  virbr2 any     anywhere             anywhere             reject-with icmp-port-unreachable

Chain OUTPUT (policy ACCEPT 206K packets, 14M bytes)
 pkts bytes target     prot opt in     out     source               destination         
    6  2022 ACCEPT     udp  --  any    virbr1  anywhere             anywhere             udp dpt:bootpc
   18  6135 ACCEPT     udp  --  any    virbr2  anywhere             anywhere             udp dpt:bootpc

I can't figure why only one of the networks work (vmpublicb). Both networks are defined exactly the same way. From vm2 outside network is reachable and vm2 is reachable from outside, vm1 is not. As you can see packets get sent out from vm1 but none are forwarded to it. To verify this I logged ICMP packets on other machines that were sent from vm1 and sure enough they are received but reply packets are not. Only hint of a problem I can spot is related to ARP table:

[root@localhost ~]# arp -a
? (145.120.88.100) at 52:54:00:81:b5:0b [ether] on virbr2
? (145.120.88.93) at 52:54:00:67:03:83 [ether] on virbr1
gateway (145.120.88.126) at e8:eb:34:b0:ad:01 [ether] on em2
localhost.localdomain (145.120.88.81) at <incomplete> on em2

That last line seems odd, why there would be that IP address for localhost? It is also vmpublic virbr1 interface IP address so probably it has something to do with this problem, but what?

I do not know how to debug this any further to pin point the problem and any suggestions would be greatly appreciated.

4
  • Are your x.y.z values actually the same everywhere? Or are the ones for virbr1 different to the ones for virbr2? (If so, please change one set to a different group name.) Sep 20, 2023 at 21:09
  • The reason I'm asking is that your x.y.z.64/255.255.255.192 network bound to em2 subsumes the virbr1 and virbr2 networks Sep 20, 2023 at 21:15
  • When asking network questions, it's really best not to use obfuscated addresses, because that can prevent us from accurately understanding your network configuration.
    – larsks
    Sep 20, 2023 at 21:36
  • Thanks for input. x.y.z is the same everywhere. I unobfuscated the ip addresses to make it clear. em2 IP address and gateway IP are things I can not affect to and I believe it is recommended to have gateway in the same network as the host. This setup worked before the reboot just fine and now the second virtual net works just fine... Sep 21, 2023 at 4:47

1 Answer 1

0

I was able to find a fix to the issue. I got there by comparing all the files /proc/sys/net/ipv4/conf/em2/ filesystem to a working installation (I was fortunate to have one in my disposal). There I found that /proc/sys/net/ipv4/conf/em2/proxy_arp was set to false (or 0) while on the working system it was set to true (or 1). So the problem was fixed by setting this to simply true:

echo 1 > /proc/sys/net/ipv4/conf/em2/proxy_arp

This of course not a persistent solution and the network interface configuration has to be changed to make it persistent. Why this issue manifested only with one virtual net and not with the other is a complete mystery to me and if someone can shed some light on the matter it would be great. I fear that this or other issues may still pop-up when the cause and fix are not thoroughly understood.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .