Totally open FreeBSD router in VirtualBox

Question

TL;DR - I'd like to set up a FreeBSD VM with one network card on my home LAN (192.168.1.0/24) and one on a private-internal-to-virtualbox network (10.9.9.0/24) and pass any and all traffic back and forth between the two.

Long time Linux user (Debian on servers) but only been using FreeBSD for about a day :)

Anyway, for my experimental stuff I have a virtualbox machine with 2 network interfaces - one bridged to my home LAN, one on an internal-only network. This machine is set up to be a block-nothing router, simply passing packets between eth0 and eth1 no matter source or destination. Easy enough to do with iptables -

iptables -A FORWARD -i eth0 -o eth1 -j ACCEPT
iptables -A FORWARD -i eth1 -o eth0 -j ACCEPT

But I've been trying to get this to work with pf and I'm only having partial success.

With

gateway_enable="YES"
pf_enable="YES"
pf_rules="/etc/pf.conf"

in my /etc/rc.conf and /etc/pf.conf containing

pass from em1:network to any keep state
pass from em0:network to any keep state
pass in inet proto tcp to any keep state
pass in inet proto udp to any keep state
pass out inet proto tcp to any keep state
pass out inet proto udp to any keep state

I can start a live disc vm attached only to internal and set the em1's IP as the default gateway, and be able to ping em1, ping em0, but I can't ping the host machine vbox is running on or any other machine on my LAN or connect via http, ssh, etc.

[root@bsdtest ~]# pfctl -sa
FILTER RULES:
pass in inet proto tcp all flags S/SA keep state
pass in inet proto udp all keep state
pass out inet proto tcp all flags S/SA keep state
pass out inet proto udp all keep state
pass inet from 10.9.9.0/24 to any flags S/SA keep state
pass inet from 192.168.1.0/24 to any flags S/SA keep state

STATES:
all tcp 192.168.1.90:22 <- 192.168.1.10:48102       ESTABLISHED:ESTABLISHED
all udp 192.168.1.2:53 <- 10.9.9.5:59075       NO_TRAFFIC:SINGLE
all udp 10.9.9.5:59075 -> 192.168.1.2:53       SINGLE:NO_TRAFFIC
all udp 192.168.1.2:53 <- 10.9.9.5:34207       NO_TRAFFIC:SINGLE
all udp 10.9.9.5:34207 -> 192.168.1.2:53       SINGLE:NO_TRAFFIC
all udp 192.168.1.2:53 <- 10.9.9.5:43515       NO_TRAFFIC:SINGLE
all udp 10.9.9.5:43515 -> 192.168.1.2:53       SINGLE:NO_TRAFFIC
all udp 192.168.1.2:53 <- 10.9.9.5:1636       NO_TRAFFIC:SINGLE
all udp 10.9.9.5:1636 -> 192.168.1.2:53       SINGLE:NO_TRAFFIC
all udp 192.168.1.2:53 <- 10.9.9.5:60124       NO_TRAFFIC:SINGLE
all udp 10.9.9.5:60124 -> 192.168.1.2:53       SINGLE:NO_TRAFFIC
all udp 192.168.1.2:53 <- 10.9.9.5:8866       NO_TRAFFIC:SINGLE
all udp 10.9.9.5:8866 -> 192.168.1.2:53       SINGLE:NO_TRAFFIC
all udp 192.168.1.2:53 <- 10.9.9.5:25534       NO_TRAFFIC:SINGLE
all udp 10.9.9.5:25534 -> 192.168.1.2:53       SINGLE:NO_TRAFFIC
all udp 192.168.1.2:53 <- 10.9.9.5:30141       NO_TRAFFIC:SINGLE
all udp 10.9.9.5:30141 -> 192.168.1.2:53       SINGLE:NO_TRAFFIC

INFO:
Status: Enabled for 0 days 00:08:28           Debug: Urgent

State Table                          Total             Rate
  current entries                       17               
  searches                            1990            3.9/s
  inserts                              253            0.5/s
  removals                             236            0.5/s
Counters
  match                                253            0.5/s
  bad-offset                             0            0.0/s
  fragment                               0            0.0/s
  short                                  0            0.0/s
  normalize                              0            0.0/s
  memory                                 0            0.0/s
  bad-timestamp                          0            0.0/s
  congestion                             0            0.0/s
  ip-option                              0            0.0/s
  proto-cksum                            0            0.0/s
  state-mismatch                         0            0.0/s
  state-insert                           0            0.0/s
  state-limit                            0            0.0/s
  src-limit                              0            0.0/s
  synproxy                               0            0.0/s
  map-failed                             0            0.0/s

TIMEOUTS:
tcp.first                   120s
tcp.opening                  30s
tcp.established           86400s
tcp.closing                 900s
tcp.finwait                  45s
tcp.closed                   90s
tcp.tsdiff                   30s
udp.first                    60s
udp.single                   30s
udp.multiple                 60s
icmp.first                   20s
icmp.error                   10s
other.first                  60s
other.single                 30s
other.multiple               60s
frag                         30s
interval                     10s
adaptive.start             6000 states
adaptive.end              12000 states
src.track                     0s

LIMITS:
states        hard limit    10000
src-nodes     hard limit    10000
frags         hard limit     5000
table-entries hard limit   200000

OS FINGERPRINTS:
758 fingerprints loaded
[root@bsdtest ~]# 

Any ideas? The lines regarding udp traffic to 192.168.1.2 from 10.9.9.5 (my live disc) would be for DNS to my home LAN name server, but no responses ever arrive... Here's what a http request shows -

[root@bsdtest ~]# pfctl -sa | grep 80
all tcp 192.168.1.10:80 <- 10.9.9.5:59436       CLOSED:SYN_SENT
all tcp 10.9.9.5:59436 -> 192.168.1.10:80       SYN_SENT:CLOSED
all tcp 192.168.1.10:80 <- 10.9.9.5:59438       CLOSED:SYN_SENT
all tcp 10.9.9.5:59438 -> 192.168.1.10:80       SYN_SENT:CLOSED

Ideas?

Answer 1

OK, found solution.

My /etc/rc.conf is fine the way it is...

The /etc/pf.conf needs to be

# cat /etc/pf.conf

ext_if="em0"
int_if="em1"
boxnet = $int_if:network
homenet = $ext_if:network

nat on $ext_if from $boxnet to any -> ($ext_if)
pass quick from { lo0, $boxnet, $homenet } to any keep state

Probably way too many variables, I could just use the original em0/em1. Anyway, this gives you -

[root@bsdtest ~]# pfctl -vnf /etc/pf.conf
ext_if = "em0"
int_if = "em1"
icmp_types = "echoreq"
boxnet = "em1:network"
homenet = "em0:network"
nat on em0:network inet from 10.9.9.0/24 to any -> 192.168.1.0/24
nat on em1:network inet from 192.168.1.0/24 to any -> 10.9.9.0/24
pass quick inet from 127.0.0.0/8 to any flags S/SA keep state
pass quick inet from 192.168.1.0/24 to any flags S/SA keep state

Answer 2

Some guesswork below so caveat emptor. It is however a very typical setup which confuses many people who start out by conflating routing, packet filtering (firewall) and NAT (Network Address Translation).

You do not state it clearly - but I would guess that your network looks like this:

Internet <-A-> SOHO Router <-B-> Server/workstation <-C-> VM

Your DNS server is on the B network which is 192.168.1.0/24

I am guessing that you Internet SOHO router is 192.168.1.1 and is set as default gateway for the network. This would be an extremely common setup.

You state yourself that the DNS server is on 192.168.1.2 and the servers bridged interface is 192.168.1.10. Behind that you have the network 10.9.9.0/24.

Your iptables setup will forward all packets on the interface. In practice you send all packet from one network to the other - even local packets. That is the important difference.

In your pf config you do not forward all packets on the interface. You have specified a network em1:network. We do not have the full config but I would guess that you actually have a nice and working barebones configuration. What bites you is the missing routes.

When you send a packet from the 10.9.9.0/24 it will reach the 192.168.1.0/24 net. Your server has a leg into that net so you will reach your DNS directly. But the DNS server on the B network has no clue how to reach the non-local C 10.9.9.0/24 network. All responses would then be sent to the "default router" which I guess is your SOHO router. This router also only knows where to find the 192.168.1.0/24 network (not 10.9.9.0/24) and would normally route everything to your external Internet link. In this case you are using proper private addresses - so the packet would instead be dropped as private addresses are not routed on the Internet.

The "proper" solution would be to setup a route on your SOHO router which tells it to route packets for 10.9.9.0/24 to 192.168.1.10. A decent router will allow you to do that. Many cheap SOHO routers unfortunately do not. In that case you could add the route on your DNS server to test it.

• The reason it works for you with iptables is that the response packet is seen on the eth0 interface and all packets are forwarded. All traffic on the B net is sent to the C net (and reverse). This includes traffic which could/should have stayed local. In effect you have setup a network bridge.
• The reason it does not work for you with your first pf setup is that you have specified what network you expect to see. Only one machine on the B net knows where to find the C net. This is 192.168.1.10 as it has an interface on the C net. In effect you have setup a basic firewall. Filtering is ready but you do not filter anything yet. But routing is missing.
• The reason it works in your second pf config (your own answer) is that you NAT the 10.9.9.0/24 network into the address space of 192.168.1.0/24. All traffic from the C net 10.9.9.0/24 will on the B net appear to come from 192.168.1.10. NAT should avoided whenever possible and only be used as a last resort.

If you do not need/want to filter packages - then I would advice you to not use a firewall. What you are trying to do should be handled by simple routing.

Answer 3

If you were looking to have "all open" gateway you could do that with a single rule only:

pass all allow-opts

No messing with explicit "keep state" nor with flags was needed.