3

I’ve run a SOHO domain for many years, including a Linux-based firewall/gateway with SNAT for internal clients, and offers a variety of services on the WAN which include port forwarding (DNAT). The firewall has ethernet NICs for LAN (eth0) and WAN (eth1). It works great.

I would like to modify this setup to transparently send all LAN-originated outgoing connections through an OpenVPN tunnel to an OpenVPN server acting as Internet gateway. The LAN's internal DNS and DHCP servers, file server, etc., should remain accessible to LAN clients. WAN services provided by the gateway on its static WAN IP should remain available as before. Based on my research so far, this scenario suggests the use of a split-tunnel VPN on the firewall, likely best implemented with connection marking and policy routing. The gateway will be an OpenVPN client.

Stackexchange question Route wireless LAN through OpenVPN is similar, but has no answers. Other resources for this specific problem are thin.

So far I have not succeeded in a round trip connection from a LAN client through the tunnel. My best try so far, described below, shows some connections going out the tunnel, and responses coming back but which appear not to be routed to the originating client. I have tried disabling anti spoofing which did not change the results.

In my research I've found an excellent overview of a split-tunnel VPN under Linux at https://www.htpcguides.com/force-torrent-traffic-vpn-split-tunnel-debian-8-ubuntu-16-04/, but its policy routing is based on a user ID, and does not cover the additional functionality required of a LAN gateway. Many resources exist for the details of implementing policy routing; the set including this are excellent: http://security.maruhn.com/iptables-tutorial/x9125.html and a valuable iproute2 "cheat sheet" is http://baturin.org/docs/iproute2.

Existing setup

My existing setup has a simple main routing table as follows (my ISP provides a static IP on a /30 subnet, the gateway's internal IP is 192.168.0.1 on the LAN 192.168.0.0/24):

# ip route list all
default via <isp_gw_ip> dev eth1
<isp_subnet>/30 dev eth1  proto kernel  scope link  src <ext_ip>
192.168.0.0/24 dev eth0  proto kernel  scope link  src 192.168.0.1 

The local table has 10 local and broadcast routes not shown here (could these be important?).

The firewall implements SNAT using:

iptables -t nat -A POSTROUTING -o eth1 \! -d 192.168.0.0/24 -s 192.168.0.0/24 -j SNAT —to-source <ext_ip>

So this is about as simple as it gets.

Plan

What I believe needs to be done is:

  • Mark all packets for connections that are going to be SNATed later. Mark them in PREROUTING for forwarded connections and OUTPUT for gateway-originated connections

  • Route marked packets through tun0:

    • Use tunnel table using ip rule add fwmark ... table tunnel
  • Set up gateway and default routes for tun0 using parameters received from the VPN server

    • Use ip route add ... table tunnel
    • This table should be structured just like the main table except no LAN route is needed
  • Add SNAT to use tun0 as the output device and tun0's IP address as the new source IP

Is that the correct set of changes?

How far I have gotten

When I start the OpenVPN client, the connection is initiated successfully and the client (the gateway) creates the tun0 device with this syslog entry:

/sbin/ip addr add dev tun0 local 10.95.3.110 peer 10.95.3.109

The following script is run by the OpenVPN client after tun0 is set up (shell variable $ifconfig_remote contains the gateway device IP address on the /30 tunnel subnet, $ifconfig_local is the IP address of tun0, and $dev the device name tun0):

# Set packet mark on to-be-SNATed packet (first one only)
iptables -t nat -A PREROUTING -i eth0 \! --dst 192.168.0.0/24 --source 192.168.0.0/24 -j MARK --set-mark 99

# Save packet mark to connection on first packet
iptables -t nat -A PREROUTING -j CONNMARK --save-mark

# Restore connection mark to packets
iptables -t mangle -A PREROUTING  -j CONNMARK --restore-mark

# Same for locally-generated packets
iptables -t nat -A OUTPUT \! --dst 192.168.0.0/24 -j MARK --set-mark 99
iptables -t nat -A OUTPUT -j CONNMARK --save-mark
iptables -t mangle -A OUTPUT -j CONNMARK --restore-mark

# SNAT for tun0
iptables -t nat -A POSTROUTING -o $dev \! -d 192.168.0.0/24 -s 192.168.0.0/24 -j SNAT --to-source $ifconfig_local

# Set default route for all marked packets to go out the VPN, governed by custom routing table
#
# First derive the network CIDR
network=`ipcalc $ifconfig_local/30 | grep ^Network: | awk ‘{print $2}’`

# Flush it first
ip route flush table tunnel

# Rule and routes
ip rule add fwmark 99 table tunnel
ip route add $network dev $dev table tunnel
ip route add default via $ifconfig_remote dev $dev table tunnel

# Flush route cache (unnecessary in recent kernels?)
ip route flush cache

The OpenVPN client reports that the initialization sequence completed with no warnings or errors.

Results

ifconfig shows a small number of packets traveling through tun0, both Tx and Rx. traceroute 4.2.2.2 from the gateway shows that packets are still going out the normal ISP path through eth1, so the policy routing for the OUTPUT chain-based marking may not be working. But ping 4.2.2.2 seems to show that only the first packet returns; subsequent packets are not returned. LAN clients are not able to connect externally at all; a ping to an external IP address results in a timeout.

tcpdump -i tun0 -vvv shows certain traffic is going through the tunnel, such as:

11:05:48.610910 IP (tos 0x0, ttl 63, id 14981, offset 0, flags [DF], proto ICMP (1), length 84)
    10.95.3.110 > b.resolvers.Level3.net: ICMP echo request, id 13207, seq 29, length 64
11:05:48.623063 IP (tos 0x0, ttl 57, id 9307, offset 0, flags [none], proto ICMP (1), length 84)
    b.resolvers.Level3.net > 10.95.3.110: ICMP echo reply, id 13207, seq 29, length 64

which implies either a ping or traceroute appears to be successfully routed through the tunnel. But it may not be getting back to the originating client? I also see DNS traffic similarly, and a few others.

Additional diagnostic info

ip route list main shows one additional route, involving tun0, has been added by the VPN startup in the main table, is this OK?

default via <isp_gw_ip> dev eth1 
10.95.3.109 dev tun0  proto kernel  scope link  src 10.95.3.110 
<isp_subnet>/30 dev eth1  proto kernel  scope link  src <ext_ip>
192.168.0.0/24 dev eth0  proto kernel  scope link  src 192.168.0.1 

iptables -L -t mangle shows

Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         
CONNMARK   all  --  anywhere             anywhere             CONNMARK restore

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
CONNMARK   all  --  anywhere             anywhere            CONNMARK restore

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         

iptables -L -t nat shows

Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         
NAT_PREROUTING_CHAIN  all  --  anywhere             anywhere            
POST_NAT_PREROUTING_CHAIN  all  --  anywhere             anywhere           
MARK       all  --  192.168.0.0/24      !192.168.0.0/24       MARK set 0x63
CONNMARK   all  --  anywhere             anywhere             CONNMARK save

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
MARK       all  --  anywhere            !192.168.0.0          MARK set 0x63
CONNMARK   all  --  anywhere             anywhere            CONNMARK save

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         
TCPMSS     tcp  --  anywhere             anywhere             tcp flags:SYN,RST/SYN TCPMSS clamp to PMTU
NAT_POSTROUTING_CHAIN  all  --  anywhere             anywhere            
SNAT       all  --  192.168.0.0/24      !192.168.0.0/24       to:<ext_ip>
POST_NAT_POSTROUTING_CHAIN  all  --  anywhere             anywhere            
SNAT       all  --  192.168.0.0/24      !192.168.0.0/24       to:10.95.3.110

Chain NAT_POSTROUTING_CHAIN (1 references)
target     prot opt source               destination         

Chain NAT_PREROUTING_CHAIN (1 references)
target     prot opt source               destination         

Chain POST_NAT_POSTROUTING_CHAIN (1 references)
target     prot opt source               destination         

Chain POST_NAT_PREROUTING_CHAIN (1 references)
target     prot opt source               destination          

I am disabling the default routes that the OpenVPN server pushes to the gateway using route-noexec in the OpenVPN config file.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.