Virtual router in homelab Proxmox VM: how to send requests from main/home network to VM behind virtual router?

Question

I have a homelab setup like this (diagram is also available here):

One thing to note here is, my Proxmox host just has 1 NIC for ethernet cable. It has WiFi too but I decided not to use it because it won't play nicely with what I wanted to do (that's my understanding after reading dozens of articles/forums about it).

So I plugged an ethernet cable from a WiFi repeater. This Proxmox host is located in different place in my home - that's why I didn't plug the ethernet directly to the home router. From my laptop, I can wirelessly SSH into the Proxmox host (192.168.1.5) just fine. I can also wirelessly access Proxmox's web UI at https://192.168.1.5:8006 just fine.

What I want to do here is network segmentation between my home network and the VMs network. I know there are free router software out there like pfSense, OpenWrt and etc. But I want to take this the opportunity to build it from scratch so I can better understand how things are working together.

I'm planning to run some VMs in this Proxmox host. They all should run within the VMs network I'm creating here. Some VMs may run on dynamic IP/DHCP mode and some may run on static IP mode. 99% of them will be Linux VMs. I'm good without IPv6 support in this VMs network for now. Also, using VLANs is not an option because my router doesn't support VLANs.

With that said, I followed this guide to create a virtual router in Ubuntu VM. The only thing I did differently from that guide is I created a bridge (br0) and assign it a static IP (192.168.2.1/24) so it can be a router / virtual switch for my VMs network.

I didn't follow the Port Forwarding from the Internet to the LAN and So, about that testing... steps because I don't need them. In other words, I followed all the steps until and including Allowing traffic out to the Internet step.

What I see currently:

[OK] VMs can ping each other
[OK] VMs can ping internet
[Not OK] VM 1 and VM 2 can ping devices in home network e.g. laptop and router
[OK] Laptop can ping Router VM
[Not OK] Laptop can't ping VM 1 or VM 2*

* I already added a static route in my laptop by following this guide. I also tried adding a static route in the home router/gateway.

FWIW, below are my configs...

ip a from Proxmox host:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr0 state UP group default qlen 1000
    link/ether fc:4d:d4:f3:fe:d3 brd ff:ff:ff:ff:ff:ff
    altname enp0s25
3: wlp2s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 28:c2:dd:a5:46:53 brd ff:ff:ff:ff:ff:ff
4: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether fc:4d:d4:f3:fe:d3 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.5/24 scope global vmbr0
       valid_lft forever preferred_lft forever
    inet6 fe80::fe4d:d4ff:fef3:fed3/64 scope link
       valid_lft forever preferred_lft forever
5: vmbr1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 8e:7e:21:7e:f9:e1 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::8c7e:21ff:fe7e:f9e1/64 scope link
       valid_lft forever preferred_lft forever
11: tap104i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr0 state UNKNOWN group default qlen 1000
    link/ether b6:5c:75:be:38:d5 brd ff:ff:ff:ff:ff:ff
12: vmbr2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 3e:6b:04:2c:d9:e0 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::3c6b:4ff:fe2c:d9e0/64 scope link
       valid_lft forever preferred_lft forever
13: tap104i1: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr2 state UNKNOWN group default qlen 1000
    link/ether 46:d9:8f:dd:ab:77 brd ff:ff:ff:ff:ff:ff
23: tap102i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr2 state UNKNOWN group default qlen 1000
    link/ether 46:9c:a4:fd:81:27 brd ff:ff:ff:ff:ff:ff
24: tap103i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr2 state UNKNOWN group default qlen 1000
    link/ether 12:90:ee:36:27:7a brd ff:ff:ff:ff:ff:ff

Content of /etc/network/interfaces in Proxmox host:

auto lo
iface lo inet loopback

iface eno1 inet manual

auto vmbr0
iface vmbr0 inet static
    address 192.168.1.5/24
    gateway 192.168.1.1
    bridge-ports eno1
    bridge-stp off
    bridge-fd 0

iface wlp2s0 inet manual

auto vmbr1
iface vmbr1 inet manual
    bridge-ports none
    bridge-stp off
    bridge-fd 0

auto vmbr2
iface vmbr2 inet manual
    bridge-ports none
    bridge-stp off
    bridge-fd 0

post-up echo 1 > /proc/sys/net/ipv4/ip_forward
post-up iptables -t nat -A POSTROUTING -s '192.168.2.0/24' -o vmbr0 -j MASQUERADE
post-down iptables -t nat -D POSTROUTING -s '192.168.2.0/24' -o vmbr0 -j MASQUERADE

Notes:

vmbr1 was from my previous experiment and not relevant/related here; none of my VMs are using vmbr1 network
The post-up and post-down lines were copied from this guide - part of my trial and error effort

ip a from Router VM:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 2a:06:12:a9:18:98 brd ff:ff:ff:ff:ff:ff
    altname enp0s18
    inet 192.168.1.175/24 metric 100 brd 192.168.1.255 scope global dynamic ens18
       valid_lft 4270sec preferred_lft 4270sec
    inet6 fe80::2806:12ff:fea9:1898/64 scope link
       valid_lft forever preferred_lft forever
3: ens19: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master br0 state UP group default qlen 1000
    link/ether ca:bc:10:17:e0:2f brd ff:ff:ff:ff:ff:ff
    altname enp0s19
    inet6 fe80::c8bc:10ff:fe17:e02f/64 scope link
       valid_lft forever preferred_lft forever
4: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether da:56:23:09:09:72 brd ff:ff:ff:ff:ff:ff
    inet 192.168.2.1/24 brd 192.168.2.255 scope global br0
       valid_lft forever preferred_lft forever
    inet6 fe80::d856:23ff:fe09:972/64 scope link
       valid_lft forever preferred_lft forever

Content of /etc/network/interfaces in Router VM:

auto lo
iface lo inet loopback

auto ens18
iface ens18 inet dhcp

#auto ens19
#iface ens19 inet static
#   address 192.168.99.1
#   netmask 255.255.255.0

IP forward config in Router VM:

$ cat /etc/sysctl.conf | grep net.ipv4.ip_forward
net.ipv4.ip_forward=1

Content of /etc/network/if-pre-up.d/iptables in Router VM:

#!/bin/sh
/sbin/iptables-restore < /etc/network/iptables

Content of /etc/network/iptables in Router VM:

*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]

# ens18 is WAN interface
# br0 is LAN interface
-A POSTROUTING -o ens18 -j MASQUERADE

COMMIT

*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]

# Service rules
# basic global accept rules - ICMP, loopback, traceroute, established all accepted
-A INPUT -s 127.0.0.0/8 -d 127.0.0.0/8 -i lo -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -m state --state ESTABLISHED -j ACCEPT

# enable traceroute rejections to get sent out
-A INPUT -p udp -m udp --dport 33434:33523 -j REJECT --reject-with icmp-port-unreachable

# DNS - accept from LAN
-A INPUT -i br0 -p tcp --dport 53 -j ACCEPT
-A INPUT -i br0 -p udp --dport 53 -j ACCEPT

# DHCP client requests - accept from LAN
-A INPUT -i br0 -p udp --dport 67:68 -j ACCEPT

# SSH - accept from WAN
-A INPUT -i ens18 -p tcp --dport 22 -j ACCEPT

# drop all other inbound traffic
-A INPUT -j DROP

# Forwarding rules
# forward packets along established/related connections
-A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT

# forward ICMP
-A FORWARD -p icmp -j ACCEPT
-A FORWARD -p icmp -s 0.0.0.0/0 -d 192.168.2.0/24 -j ACCEPT

# forward from LAN (br0) to WAN (ens18)
-A FORWARD -i br0 -o ens18 -j ACCEPT

# drop all other forwarded traffic
-A FORWARD -j DROP

COMMIT

Result of sudo iptables -S command in Router VM:

-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-A INPUT -s 127.0.0.0/8 -d 127.0.0.0/8 -i lo -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -m state --state ESTABLISHED -j ACCEPT
-A INPUT -p udp -m udp --dport 33434:33523 -j REJECT --reject-with icmp-port-unreachable
-A INPUT -i br0 -p tcp -m tcp --dport 53 -j ACCEPT
-A INPUT -i br0 -p udp -m udp --dport 53 -j ACCEPT
-A INPUT -i br0 -p udp -m udp --dport 67:68 -j ACCEPT
-A INPUT -i ens18 -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT -j DROP
-A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -p icmp -j ACCEPT
-A FORWARD -d 192.168.2.0/24 -p icmp -j ACCEPT
-A FORWARD -i br0 -o ens18 -j ACCEPT
-A FORWARD -j DROP

DHCP config in /etc/dhcp/dhcpd.conf in Router VM:

subnet 192.168.2.0 netmask 255.255.255.0 {
    range 192.168.2.100 192.168.2.199;
    option routers 192.168.2.1;
    option domain-name-servers 192.168.2.1;
    option broadcast-address 192.168.2.255;
        host ubuntu-desktop {
          hardware ethernet 5A:5B:74:01:69:34;
          fixed-address 192.168.2.2;
        }
}

Note: ubuntu-desktop is VM 1

Questions:

How can I block VM 1 and VM 2 from accessing devices in my home network? TCP/UDP/ICMP and everything.
How can I access the VM 1 and VM 2 when I'm using laptop or desktop e.g. sending ping/ICMP request (ping 192.168.2.2/ping 192.168.2.101), access Nginx process running on port 80 (curl 192.168.2.2/curl 192.168.2.101) and SSH server process running on port 22 (ssh user@192.168.2.2/ssh user@192.168.2.101) on both VMs?

I did some research about this and I got into DNAT/SNAT/Masquerade topic but I'm not sure if that's what I need. If yes, what the rules should look like? I'm new to this iptables topic. I tried rules below but I still can't ping to the VM 1 and VM 2 from my laptop.

iptables -I FORWARD -s 192.168.1.0/24 -d 192.168.2.0/24 -j ACCEPT
iptables -I FORWARD -s 192.168.2.0/24 -d 192.168.1.0/24 -m conntrack --ctstate ESTABLISHED -j ACCEPT
iptables -t nat -I POSTROUTING -s 192.168.1.0/24 -d 192.168.2.0/24 -j SNAT --to 192.168.2.1

What you've implemented is a NAT/Router. You don't want any of the NAT components, which means throwing away much of the article''s instructions. At that point I'm not sure the article remains relevant for you — roaima, Dec 29, 2022 at 22:18
Do you have any other relevant guides/tutorials/resources for me? — Zulhilmi Zainudin, Dec 30, 2022 at 10:50

Bruce Malaudzi · Accepted Answer · 2022-12-29 21:17:27Z

0

Bridge interface allows VM1 and VM2 to ping laptop and PC since they belong to the same broadcast domain (192.168.2.0/24). You will need to get rid of bridge interface and replace with host-only adapter / network.

Lets assume that your VM isolated network is 192.168.10.0/24. You must then enable masquerade on router VM

$ sudo iptables -A POSTROUTING -o ens18 -j MASQUERADE

For laptop/PC to ping VM1 or VM2, you will need to add a static route on the physical GW/Router via promox (192.168.1.5).

$ sudo ip route add 192.168.10.0/24 via 192.168.1.5

If your physical router/gw does not support static routes, you might have issues with routing. Above command is Linux specific but I am using it to illustrate the requirement.

Reference: https://dannyda.com/2020/06/01/how-to-create-an-internal-only-isolated-network-for-guest-os-virtual-machines-vm-on-proxmox-ve-pve-like-in-vmware-workstation-host-only-network-but-different/

answered Dec 29, 2022 at 21:17

Bruce Malaudzi

1,4101 gold badge4 silver badges9 bronze badges

"Bridge interface allows VM1 and VM2 to ping laptop and PC since they belong to the same broadcast domain (192.168.2.0/24)" - they here refers to VM1 and VM2 yes? Or, did you refer to to VM1, VM2, laptop and PC? Because VM1 and VM2 are on 192.168.2.0/24 network while laptop and PC and on 192.168.1.0/24 network.
– Zulhilmi Zainudin
Dec 30, 2022 at 10:39
I looked at the reference link (dannyda.com site) but that's not what I wanted to do. The VM1 and VM2 needs to have access to internet via my router at 192.168.1.1. If I get the article correctly, VMs in the isolated network can't reach internet, correct? If yes, how can I make an isolated network (using dannyda.com approach) and give my VMs internet access?
– Zulhilmi Zainudin
Dec 30, 2022 at 10:42
WRT static route, as I mentioned in my question ("I also tried adding a static route in the home router/gateway"), I can see the ping request from my laptop (192.168.1.10) to the VM1 (192.168.2.2) when I ran tcpdump -n -i any icmp in the PVE host. But it stopped there. No packets were sent back from VM1 (192.168.2.2) to my laptop (192.168.1.10).
– Zulhilmi Zainudin
Dec 30, 2022 at 10:49

Add a comment |

Stack Exchange Network

Virtual router in homelab Proxmox VM: how to send requests from main/home network to VM behind virtual router?

1 Answer 1

Your Answer

Not the answer you're looking for? Browse other questions tagged
networking
iptables
virtual-machine
bridge
proxmox
or ask your own question.

Hot Network Questions

Virtual router in homelab Proxmox VM: how to send requests from main/home network to VM behind virtual router?

1 Answer 1

Your Answer

Sign up or log in

Post as a guest

Not the answer you're looking for? Browse other questions tagged networkingiptablesvirtual-machinebridgeproxmox or ask your own question.

Related

Hot Network Questions

Not the answer you're looking for? Browse other questions tagged
networking
iptables
virtual-machine
bridge
proxmox
or ask your own question.