2

Having the following topology:

hostA(virbr0: 192.168.122.1) -- TAP interface -- (eth0: 192.168.122.85) gw (eth1: 192.168.3.51) --- (LAN: 192.168.3.0/24)

gw : QEMU VM Guest, that is acting as a gateway for other QEMU VM Guests, which are connected to it via eth1 interface. gw is connected to the QEMU Host via the Linux TAP Interface, using interface eth0. The TAB Interface is assigning IP to the eth0 dynamically.

hostA : QEMU VM Host

root@hostA:~$ ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.16.254.134  netmask 255.255.255.0  broadcast 172.16.254.255
        inet6 fe80::20c:29ff:fe25:1670  prefixlen 64  scopeid 0x20<link>
        ether 00:0c:29:25:16:70  txqueuelen 1000  (Ethernet)
        RX packets 18116  bytes 1361094 (1.3 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 14579  bytes 5334119 (5.3 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

gns3tap0-0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::50bd:d3ff:fe1a:e55  prefixlen 64  scopeid 0x20<link>
        ether 52:bd:d3:1a:0e:55  txqueuelen 1000  (Ethernet)
        RX packets 353  bytes 28543 (28.5 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 3348  bytes 183399 (183.3 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 662311  bytes 99203163 (99.2 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 662311  bytes 99203163 (99.2 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

virbr0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.122.1  netmask 255.255.255.0  broadcast 192.168.122.255
        ether 52:54:00:62:0a:c5  txqueuelen 1000  (Ethernet)
        RX packets 353  bytes 23601 (23.6 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 379  bytes 28629 (28.6 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
gw:~# ifconfig
eth0      Link encap:Ethernet  HWaddr 0C:FE:27:0C:E4:00
          inet addr:192.168.122.85  Bcast:192.168.122.255  Mask:255.255.255.0
          inet6 addr: fe80::efe:27ff:fe0c:e400/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:379 errors:3150 dropped:0 overruns:0 frame:3150
          TX packets:353 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:31851 (31.1 KiB)  TX bytes:28543 (27.8 KiB)

eth1      Link encap:Ethernet  HWaddr 0C:FE:27:0C:E4:01
          inet addr:192.168.3.51  Bcast:0.0.0.0  Mask:255.255.255.0
          inet6 addr: fe80::efe:27ff:fe0c:e401/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:302 errors:0 dropped:0 overruns:0 frame:0
          TX packets:300 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:25796 (25.1 KiB)  TX bytes:24782 (24.2 KiB)
gw:~# netstat -nr
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         192.168.122.1   0.0.0.0         UG        0 0          0 eth0
192.168.3.0     0.0.0.0         255.255.255.0   U         0 0          0 eth1
192.168.122.0   0.0.0.0         255.255.255.0   U         0 0          0 eth0

The goal to have a reachability from hostA to the VMs of the internal LAN, 192.168.3.0/24

The following routing configuration is working:

root@hostA:~$ sudo route add -net 192.168.3.0 netmask 255.255.255.0 gw 192.168.122.85
root@hostA:~$ netstat -nr
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         172.16.254.2    0.0.0.0         UG        0 0          0 eth0
172.16.254.0    0.0.0.0         255.255.255.0   U         0 0          0 eth0
172.17.0.0      0.0.0.0         255.255.0.0     U         0 0          0 docker0
192.168.3.0     192.168.122.85  255.255.255.0   UG        0 0          0 virbr0
192.168.122.0   0.0.0.0         255.255.255.0   U         0 0          0 virbr0
root@hostA:~$ ping 192.168.3.102
PING 192.168.3.102 (192.168.3.102) 56(84) bytes of data.
64 bytes from 192.168.3.102: icmp_seq=1 ttl=63 time=3.07 ms
^C
--- 192.168.3.102 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 3.071/3.071/3.071/0.000 ms
root@hostA:~$ ping 192.168.3.103
PING 192.168.3.103 (192.168.3.103) 56(84) bytes of data.
64 bytes from 192.168.3.103: icmp_seq=1 ttl=63 time=3.03 ms
64 bytes from 192.168.3.103: icmp_seq=2 ttl=63 time=2.99 ms
^C
--- 192.168.3.103 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 2.997/3.014/3.032/0.057 ms

However, since, the Linux TAP is assigning the IPs to the eth0 dynamically, it is needed to have the route, that will not use the IP Address of the next router as the next hop, but, the local IP of the exit interface, similar to the following:


root@hostA:~$ route add -net 192.168.0.0 netmask 255.255.0.0 gw 192.168.122.1 dev virbr0
root@hostA:~$ netstat -nr
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         172.16.254.2    0.0.0.0         UG        0 0          0 eth0
172.16.254.0    0.0.0.0         255.255.255.0   U         0 0          0 eth0
172.17.0.0      0.0.0.0         255.255.0.0     U         0 0          0 docker0
192.168.0.0     192.168.122.1   255.255.0.0     UG        0 0          0 virbr0
192.168.122.0   0.0.0.0         255.255.255.0   U         0 0          0 virbr

root@hostA:~$ ping -c 1 192.168.122.85
PING 192.168.122.85 (192.168.122.85) 56(84) bytes of data.
64 bytes from 192.168.122.85: icmp_seq=1 ttl=64 time=0.585 ms

--- 192.168.122.85 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.585/0.585/0.585/0.000 ms
root@hostA:~$ ping -c 1 192.168.3.51
PING 192.168.3.51 (192.168.3.51) 56(84) bytes of data.
64 bytes from 192.168.3.51: icmp_seq=1 ttl=64 time=0.956 ms

--- 192.168.3.51 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.956/0.956/0.956/0.000 ms

root@hostA:~$ ping -c 1 192.168.3.102
PING 192.168.3.102 (192.168.3.102) 56(84) bytes of data.
From 192.168.122.1 icmp_seq=1 Destination Host Unreachable

--- 192.168.3.102 ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms

root@hostA:~$ ping -c 1 192.168.3.103
PING 192.168.3.103 (192.168.3.103) 56(84) bytes of data.
From 192.168.122.1 icmp_seq=1 Destination Host Unreachable

--- 192.168.3.103 ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms


As seen, when using above route configuration, the hostA is able to reach the eth1 interface, but, not able to reach any other VMs in the internal LAN (192.168.3.0/24). However, the internal LAN hosts itself are able to reach the hostA.

Here is the iptables configs:

root@hostA:~$ sudo iptables -nvL FORWARD
Chain FORWARD (policy ACCEPT 21 packets, 1624 bytes)
 pkts bytes target     prot opt in     out     source               destination
  197 15616 DOCKER-USER  all  --  *      *       0.0.0.0/0            0.0.0.0/0
  197 15616 DOCKER-ISOLATION-STAGE-1  all  --  *      *       0.0.0.0/0            0.0.0.0/0
    0     0 ACCEPT     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
    0     0 DOCKER     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0
    0     0 ACCEPT     all  --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0
    0     0 ACCEPT     all  --  docker0 docker0  0.0.0.0/0            0.0.0.0/0
   88  7346 ACCEPT     all  --  *      virbr0  0.0.0.0/0            192.168.122.0/24     ctstate RELATED,ESTABLISHED
   88  6646 ACCEPT     all  --  virbr0 *       192.168.122.0/24     0.0.0.0/0
    0     0 ACCEPT     all  --  virbr0 virbr0  0.0.0.0/0            0.0.0.0/0

gw:~# iptables -nvL FORWARD
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 ACCEPT     all  --  eth1   eth0    192.168.122.0/24     0.0.0.0/0            ctstate NEW
    0     0 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
gw:~# sysctl -a | grep ipv4.ip_forward
net.ipv4.ip_forward = 1

dns:~# ifconfig
eth0      Link encap:Ethernet  HWaddr 0C:FE:27:0C:76:00
          inet addr:192.168.3.103  Bcast:0.0.0.0  Mask:255.255.255.0
          inet6 addr: fe80::efe:27ff:fe0c:7600/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:18638 errors:0 dropped:0 overruns:0 frame:0
          TX packets:18634 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1543312 (1.4 MiB)  TX bytes:1705846 (1.6 MiB)

dns:~# netstat -nr
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         192.168.3.51    0.0.0.0         UG        0 0          0 eth0
192.168.3.0     0.0.0.0         255.255.255.0   U         0 0          0 eth0

Questions:

  1. Is there something missing for the configuration?

  2. Is there a way to configure the working route entry, that will use the local IP of the exit interface, as the next-hop IP?

1 Answer 1

1

There are two or three separate problems.

dynamic gw IP

It would be overkill to use a routing protocol for this (unless you are using one already for other reasons). I would try to change the configuration of gw/eth0 to static. If that is not possible then you could create a user gw on hostA which gets sudo permission for setting the route to that network. Every time gw gets a new IP it could connect via SSH to hostA and run that command (maybe via ForcedCommand).

routing on gw

Routing is not configured (correctly) on gw. You must enable it at all via net.ipv4.ip_forward = 0 (man sysctl) and allow the respective connections in the firewall, see iptables -nvL FORWARD.

routing on the VMs in 192.168.3.0

The VMs which are connected to gw/eth1 must have gw configured as default gateway. If that is not the case then gw must do SNAT towards this network (iptables).

1
  • 1) dynamic gw IP The problem here is, that, I didn't really get on the way, how the TAP device is working in Linux, however, as soon as I am trying to leave the config as it is now, but, set the static IP on gw, I am no more able to reach the outside network, connected to hostA (a.k.a, Internet) 2) routing on gw Did you mean, that in order to enable routing, it is needed to set net.ipv4.ip_forward = 1, isn't it? Have attached, the iptables and sysctl output, showing the settings are enabled 3) Right, the same is configured on all VMs in 192.168.3.0, attached the output
    – readonly
    Jul 14, 2020 at 20:45

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .