0

Advanced info

Machines involved:
A := vpn server inet 10.9.0.1 peer 10.9.0.2 tun0
B := vpn server inet 10.8.0.1 peer 10.9.0.2 tun2
     vpn client inet 10.9.0.6 peer 10.9.0.5 tun1
C := vpn client inet 10.8.0.6 peer 10.9.0.5 tun0

Description

There are three machines. Machine B runs both an openvpn server and client. Now I when machine A pushes redirect-gateway def1 to machine B, then machine C can't connect to the openvpn-server on machine B obviously, because the vpn-server reply packets are routed over to machine A, due to the 0.0.0.0/1 via 10.9.0.5, 128.0.0.0/1 via 10.9.0.5 routes. Now I need policy routing to route the packets over machine's B WAN gateway.

Failed attempt 1

I tried to mark the packets that are sent from the vpn server with:

iptables -t mangle -A OUTPUT -p tcp -m multiport --sports 1194 -j MARK --set-xmark 14
ip route add table 14 default via WAN_IP
ip rule add fwmark 14 table 14

But they are still routed over to machine A, which I could observe with tcpdump -i tun1 on machine B and tcpdump -i tun0 on machine A.

**machine B**
eth0:
22:22:33.734511 IP machine_C_IP.54089 > machine_B_IP.1194: UDP, length 86
tun1:
22:32:42.713371 IP 10.9.0.6.1194 > machine_C_IP.34514: UDP, length 94

**machine A**
tun0:
20:47:47.769628 IP machine_A_IP.openvpn > machine_C_IP.40287: UDP, length 94

Failed attempt 2

I also tried to mark packets by the owner of the openvpn process. I added to openvpn@.service Group = vpnserver and then:

iptables -t mangle -A OUTPUT -m owner --gid-owner vpnserver -j MARK --set-xmark 14 ip route add table 14 default via WAN_GATEWAY ip rule add fwmark 14 table 14

But packets are still routed over to machine A.

Working attempt

Following works, but it's something I want to avoid using, because it's not convenient, because I would have to add routes for every client by its IP. Some clients have also dynamic IPs, thus I would have to use a script to retrieve the right IP by a domain name.

ip route add table 14 default via WAN_IP
ip rule add to machine_C_IP table 14

Abstract

Maybe the marked packets fail being routed rightly, because the iptables' OUTPUT chain applies after the actual routing? Or do I miss something while marking the packets? Is there any other way to route the outgoing vpn server packets to my WAN gateway?

Current state

Routes
default via WAN_GATEWAY dev eth0  table 42 
0.0.0.0/1 via 10.9.0.5 dev tun1 
default via WAN_GATEWAY dev eth0 onlink 
10.1.1.0/24 dev veth0  proto kernel  scope link  src 10.1.1.2 
10.8.0.0/24 via 10.8.0.2 dev tun0 
10.8.0.2 dev tun0  proto kernel  scope link  src 10.8.0.1 
10.9.0.0/24 via 10.9.0.5 dev tun1 
10.9.0.5 dev tun1  proto kernel  scope link  src 10.9.0.6 
WAN_SUBNET/26 via WAN_GATEWAY dev eth0 
WAN_SUBNET/26 dev eth0  proto kernel  scope link  src WAN_IP 
MACHINE_A_IP via WAN_GATEWAY dev eth0 
128.0.0.0/1 via 10.9.0.5 dev tun1
IP rules
0:      from all lookup local 
32760:  from all fwmark 0xe lookup 42 
32761:  from all fwmark 0xa lookup 42 
32764:  from all to WAN_IP lookup 42 
32765:  from WAN_IP lookup 42 
32766:  from all lookup main 
32767:  from all lookup default 
  • 1
    What kind of routing do you require for the client running the server? Only routing from its clients to the other vpn? – sebasth Sep 13 '17 at 17:37
  • Machine B should route packets from 10.8.0.0/24 back over the WAN gateway to its clients directly and not over the other vpn server on machine A (10.9.0.1). Machine's B vpn client connects also directly to machine's A vpn server and machine's C vpn client connects directly to machine's B vpn server. – fremon Sep 13 '17 at 17:44
  • It's the same scenario that you explained here unix.stackexchange.com/a/391368/108542 , just that machine B now also runs an openvpn-server, not just vpn client. Again only outgoing traffic is routed over machine's A vpn server, but now with the exception for outgoing packets of the newly added vpn server of machine B. which I could only accomplish with ip rule add to machine_C_IP table 14. Didn't find a more clever way yet. – fremon Sep 14 '17 at 10:17
  • 1
    In short, routes you want on B: from C forward to WAN and from B to A, except openvpn? Also, why you using --set-xmark (and not --set-mark), and did you flush the routing caches after adding the routing rules? – sebasth Sep 14 '17 at 14:24
  • 1
    I've actually never tried this, but if you interested: (1) Configure your OpenVPN instances to share the same primary group (create a group openvpn). Create iptables rule to match that group iptables -t mangle -A OUTPUT -m owner --gid openvpn -j MARK --mark 42. (2) Use a routing rule like before to use a separate WAN routing table for marked traffic (ip rule add ... like before). Now you should be able to connect with client on C. (3) ip rule add iff tun2 table 42. – sebasth Sep 14 '17 at 20:29
1

Building on a previous answer in the thread, this is how I was able to solve the issue on my Ubuntu 17.10 (artful) computer running OpenVPN 2.4.3.

1) Disable reverse path filtering

We need to change unicast reverse path forwarding to loose mode (rp_filter=2) instead of strict mode (rp_filter=1). In strict mode, an incoming packet must be received on the same network interface that would be used to forward the return packet.

sysctl -w net.ipv4.conf.<if>.rp_filter=2

Where <if> is the physical ethernet interface the computer uses to connect to the router. In my case, the command looked like this: sysctl -w net.ipv4.conf.enp2s0.rp_filter=2. To read the kernel parameter, use sysctl net.ipv4.conf.<if>.rp_filter.

To make the change persistent after a reboot, add or change this line in /etc/sysctl.conf:

net.ipv4.conf.<if>.rp_filter=2

2) Set OpenVPN server to mark encrypted packets

In the server .conf file, add:

mark <value>

Where <value> is an arbitrarely unique value for marking the packets. This will allow rerouting the packets encrypted by the OpenVPN server. I used mark 9.

Restart the OpenVPN server.

3) Reroute marked packets to the router with a new ip rule and ip route

ip rule add fwmark <value> table <N>
ip route add default via <router> table <N>

Where <N> is an arbitrarely unique number for the routing table and <router> is the router IP address in my LAN. In other words, I executed these commands: ip rule add fwmark 9 table 42 and ip route add default via 192.168.8.1 table 42.

Making these changes persistent after reboot is more tricky, especially with Ubuntu 17.10 that uses the new netplan package for managing network connections instead of the traditionnal /etc/network/interfaces file. For lack of a better solution, I added a series of commands in the OpenVPN systemd service file. This will create the ip rule and route before the OpenVPN is started: ExecStartPre=-/sbin/ip rule add fwmark 9 table 42 and ExecStartPre=-/sbin/ip route add default via 192.168.8.1 table 42. I use the minus - sign before the command path to make sure the server will start even if the command fails (in case the rule or route already exists), see the systemd.service man page for more information.

Tip: I run the OpenVPN server as a systemd service on my computer, the best way to troubleshoot this issue in my case was to increase the verbosity of the OpenVPN server in the .conf file to 6, and then to use journalctl -fu openvpn-server@<conf_file_name>.service to see if the connection could be established. If reverse path filtering is not disabled, you may not even see any activity when the clients try to connect. After setting the rp_filter to 2 (step 1), I could see some activity, but the TLS handshake would fail. Once I managed to reroute the OpenVPN packets to the router (step 2-3), my clients were able to connect.

0

Why it fails

Failed attempt 1 and Failed attempt 2 don't work, because the iptables' chain runs after the routing decision, see the iptables' diagramm http://inai.de/images/nf-packet-flow.png.

Besides that, Failed attempt 2 uses a wrong approach to set the User/Group of the vpn server instance. You need to specify the user or group in /etc/openvpn/server.conf and not in the systemd unit file. I.e.:

  1. Create the user or group adduser vpnserver
  2. Then add user or group to /etc/openvpn/server.conf user vpnserver
  3. Verify the vpn server process runs as that user/group with ps -A o pid,cmd,user,group|grep vpn

Solution

You can mark packets of local processes only if the processes set a socket option. OpenVPN supports that: https://community.openvpn.net/openvpn/wiki/Openvpn23ManPage

You can mark a packet with the option --mark value in /etc/openvpn/server.conf and then use ip rule add fwmark value table 42 with ip routes default via WAN_GATEWAY table 42.

Machine C can then connect to machine's B vpn server.

Alternative

Alternatively you could use netns namespaces for the vpn server and client and then make use of iptables' chains.

  • The statement "the iptables' chain runs after the routing decision" is wrong, obviously wrong. Even the diagram shows two points with routing decisions. Probably because you can use the output device in iptables rules. But the diagram is wrong. You can do DNAT in the nat OUTPUT chain and obviously the routing decision cannot be made before the target address is known. – Hauke Laging Sep 16 '17 at 21:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.