How do I route all traffic on a machine over an openvpn server while the machine itself runs an openvpn server?

Question

Advanced info

Machines involved:
A := vpn server inet 10.9.0.1 peer 10.9.0.2 tun0
B := vpn server inet 10.8.0.1 peer 10.9.0.2 tun2
     vpn client inet 10.9.0.6 peer 10.9.0.5 tun1
C := vpn client inet 10.8.0.6 peer 10.9.0.5 tun0

Description

There are three machines. Machine B runs both an openvpn server and client. Now I when machine A pushes redirect-gateway def1 to machine B, then machine C can't connect to the openvpn-server on machine B obviously, because the vpn-server reply packets are routed over to machine A, due to the 0.0.0.0/1 via 10.9.0.5, 128.0.0.0/1 via 10.9.0.5 routes. Now I need policy routing to route the packets over machine's B WAN gateway.

Failed attempt 1

I tried to mark the packets that are sent from the vpn server with:

iptables -t mangle -A OUTPUT -p tcp -m multiport --sports 1194 -j MARK --set-xmark 14
ip route add table 14 default via WAN_IP
ip rule add fwmark 14 table 14

But they are still routed over to machine A, which I could observe with tcpdump -i tun1 on machine B and tcpdump -i tun0 on machine A.

**machine B**
eth0:
22:22:33.734511 IP machine_C_IP.54089 > machine_B_IP.1194: UDP, length 86
tun1:
22:32:42.713371 IP 10.9.0.6.1194 > machine_C_IP.34514: UDP, length 94

**machine A**
tun0:
20:47:47.769628 IP machine_A_IP.openvpn > machine_C_IP.40287: UDP, length 94

Failed attempt 2

I also tried to mark packets by the owner of the openvpn process. I added to openvpn@.service Group = vpnserver and then:

iptables -t mangle -A OUTPUT -m owner --gid-owner vpnserver -j MARK --set-xmark 14 ip route add table 14 default via WAN_GATEWAY ip rule add fwmark 14 table 14

But packets are still routed over to machine A.

Working attempt

Following works, but it's something I want to avoid using, because it's not convenient, because I would have to add routes for every client by its IP. Some clients have also dynamic IPs, thus I would have to use a script to retrieve the right IP by a domain name.

ip route add table 14 default via WAN_IP
ip rule add to machine_C_IP table 14

Abstract

Maybe the marked packets fail being routed rightly, because the iptables' OUTPUT chain applies after the actual routing? Or do I miss something while marking the packets? Is there any other way to route the outgoing vpn server packets to my WAN gateway?

Current state

Routes

default via WAN_GATEWAY dev eth0  table 42 
0.0.0.0/1 via 10.9.0.5 dev tun1 
default via WAN_GATEWAY dev eth0 onlink 
10.1.1.0/24 dev veth0  proto kernel  scope link  src 10.1.1.2 
10.8.0.0/24 via 10.8.0.2 dev tun0 
10.8.0.2 dev tun0  proto kernel  scope link  src 10.8.0.1 
10.9.0.0/24 via 10.9.0.5 dev tun1 
10.9.0.5 dev tun1  proto kernel  scope link  src 10.9.0.6 
WAN_SUBNET/26 via WAN_GATEWAY dev eth0 
WAN_SUBNET/26 dev eth0  proto kernel  scope link  src WAN_IP 
MACHINE_A_IP via WAN_GATEWAY dev eth0 
128.0.0.0/1 via 10.9.0.5 dev tun1

IP rules

0:      from all lookup local 
32760:  from all fwmark 0xe lookup 42 
32761:  from all fwmark 0xa lookup 42 
32764:  from all to WAN_IP lookup 42 
32765:  from WAN_IP lookup 42 
32766:  from all lookup main 
32767:  from all lookup default 

Answer 1

Building on a previous answer in the thread, this is how I was able to solve the issue on my Ubuntu 17.10 (artful) computer running OpenVPN 2.4.3.

1) Disable reverse path filtering

We need to change unicast reverse path forwarding to loose mode (rp_filter=2) instead of strict mode (rp_filter=1). In strict mode, an incoming packet must be received on the same network interface that would be used to forward the return packet.

sysctl -w net.ipv4.conf.<if>.rp_filter=2

Where <if> is the physical ethernet interface the computer uses to connect to the router. In my case, the command looked like this: sysctl -w net.ipv4.conf.enp2s0.rp_filter=2. To read the kernel parameter, use sysctl net.ipv4.conf.<if>.rp_filter.

To make the change persistent after a reboot, add or change this line in /etc/sysctl.conf:

net.ipv4.conf.<if>.rp_filter=2

2) Set OpenVPN server to mark encrypted packets

In the server .conf file, add:

mark <value>

Where <value> is an arbitrarely unique value for marking the packets. This will allow rerouting the packets encrypted by the OpenVPN server. I used mark 9.

Restart the OpenVPN server.

3) Reroute marked packets to the router with a new ip rule and ip route

ip rule add fwmark <value> table <N>
ip route add default via <router> table <N>

Where <N> is an arbitrarely unique number for the routing table and <router> is the router IP address in my LAN. In other words, I executed these commands: ip rule add fwmark 9 table 42 and ip route add default via 192.168.8.1 table 42.

Making these changes persistent after reboot is more tricky, especially with Ubuntu 17.10 that uses the new netplan package for managing network connections instead of the traditionnal /etc/network/interfaces file. For lack of a better solution, I added a series of commands in the OpenVPN systemd service file. This will create the ip rule and route before the OpenVPN is started: ExecStartPre=-/sbin/ip rule add fwmark 9 table 42 and ExecStartPre=-/sbin/ip route add default via 192.168.8.1 table 42. I use the minus - sign before the command path to make sure the server will start even if the command fails (in case the rule or route already exists), see the systemd.service man page for more information.

Tip: I run the OpenVPN server as a systemd service on my computer, the best way to troubleshoot this issue in my case was to increase the verbosity of the OpenVPN server in the .conf file to 6, and then to use journalctl -fu openvpn-server@<conf_file_name>.service to see if the connection could be established. If reverse path filtering is not disabled, you may not even see any activity when the clients try to connect. After setting the rp_filter to 2 (step 1), I could see some activity, but the TLS handshake would fail. Once I managed to reroute the OpenVPN packets to the router (step 2-3), my clients were able to connect.

Answer 2

Why it fails

Failed attempt 1 and Failed attempt 2 don't work, because the iptables' chain runs after the routing decision, see the iptables' diagramm http://inai.de/images/nf-packet-flow.png.

Besides that, Failed attempt 2 uses a wrong approach to set the User/Group of the vpn server instance. You need to specify the user or group in /etc/openvpn/server.conf and not in the systemd unit file. I.e.:

1. Create the user or group adduser vpnserver
2. Then add user or group to /etc/openvpn/server.conf user vpnserver
3. Verify the vpn server process runs as that user/group with ps -A o pid,cmd,user,group|grep vpn

Solution

You can mark packets of local processes only if the processes set a socket option. OpenVPN supports that: https://community.openvpn.net/openvpn/wiki/Openvpn23ManPage

You can mark a packet with the option --mark value in /etc/openvpn/server.conf and then use ip rule add fwmark value table 42 with ip routes default via WAN_GATEWAY table 42.

Machine C can then connect to machine's B vpn server.

Alternative

Alternatively you could use netns namespaces for the vpn server and client and then make use of iptables' chains.