How to accommodate a VPN client, a VPN server and regular traffic at the same time on a single Linux box

Question

What I want to achieve

I am contemplating the following setup on my Linux box (a Raspberry Pi v3 under a recent brew of Raspberry Pi OS):

1. A VPN client for all the outgoing connections by default
2. A VPN server to accomodate incoming connections from the internet to my box
3. A "vanilla" interface (eth0 here) to be able to get out of my box while bypassing the deafutl VPN client

I use interface #3 to grab my public IP as delivered by my ISP, i.e. the IP of my home network. E.g. it allows me to successfully update my DNS provider account with my "real" (read non-VPN client) IP:

curl https://my.dns.provider/refresh_my_account_id_query --interface etho

                                                                                                                                                
                                                                                                                                                
                                                                                                                                                
+---------------------------------------------------------------+    +-----------------+    +---------+                                         
|                         My Linux box                          |    |                 |    |         |                                         
|                                                               |    | My home network |    | My ISP  |                                         
|                                                               |    |                 |    |         |                                         
|  +------------+                                               |    |                 |    |         |     +------------------+                
|  | VPN Client |  Dev: tun0 | IP:<pre-defined by VPN provider> |    |                 |    |         |     |  VPN Provider's  |                
|  | (default   <------------------------------------------------------------------------------------------->  Servers         <---------+      
|  | outgoing)  |                                               |    |                 |    |         |     |                  |         |      
|  +------------+                                               |    |                 |    |         |     +------------------+         |      
|                                                               |    |                 |    |         |                           +------v-----+
|  +------------+                                               |    |                 |    |         |                           |            |
|  |            |  Dev: tun1 | IP:10.8.0.1                      |    |                 |    |         |                           |  Internet  |
|  | VPN Server <----------------------------------------------------------------------------------------------------------------->            |
|  |            |                                               |    |                 |    |         |                           |            |
|  +------------+                                               |    |                 |    |         |                           |            |
|                                                               |    |                 |    |         |                           |            |
|  +------------+                                               |    |                 |    |         |                           |            |
|  |Only for    |   Dev eth0 | IP:192.168.0.200                 |    |    -------------------------------------------------------->            |
|  |explicit VPN<--------------------------------------------------------/             |    |         |                           |            |
|  |bypass      |                                               |    |                 |    |         |                           |            |
|  +------------+                                               |    |                 |    |         |                           +------------+
|                                                               |    |                 |    |         |                                         
|                                                               |    |                 |    |         |                                         
+---------------------------------------------------------------+    +-----------------+    +---------+                                         

The issues I'm facing

I can't have both VPNs (interface tun0 and tun1) working at the same time: when they are both up and configured I can't connect my VPN server from the internet anymore. All the rest works well though. It seems to me that when both the routes and rules for interfaces tun0 and tun1 are setup, all the incoming traffic just goes on one default interface, or is dropped, I'm not sure.

My current setup for each interface

Following the list of interfaces in the my setup description:

1. I have setup my VPN client using a plain openvpn config file which is being used by systemctl to run the client as a service. This work well on its own, i.e. all the traffic is going through the tun0 interface by default.
2. I have setup my VPN server using PiVPN. This was really simple and would work after just a one-shot try.
3. I am manually adding ip route default via 192.168.0.1 dev eth0 proto dhcp src 192.168.0.200 metric 202 and this is working as expected.

When booting the box, all 3 interfaces come up (on top of lo) and ip route gives:

0.0.0.0/1 via 172.94.109.161 dev tun0 
default via 172.94.109.161 dev tun0 
default via 192.168.0.1 dev eth0 proto dhcp src 192.168.0.200 metric 202 
10.8.0.0/24 dev tun1 proto kernel scope link src 10.8.0.1 
128.0.0.0/1 via 172.94.109.161 dev tun0 
172.94.109.4 via 192.168.0.1 dev eth0 
172.94.109.160/28 dev tun0 proto kernel scope link src 172.94.109.163 
192.168.0.0/24 dev eth0 proto dhcp scope link src 192.168.0.200 metric 202 

What I understand from the above is:

• Lines #1 and #2 are the same and are meant to get all my default outgoing traffic through my VPN client as I intend, I should remove one of them though.
• Line #3 is added manually by me and works as I intend to bypass my VPN client should I target directly the eth0 interface, but is it breaking something else?
• Line #4 is meant to get anything received on the VPN server's IP to be managed by tun1, my VPN server
• Line #5 to #8 I have no clue, seems like needed by my VPN client to work properly

What I am asking for

I need another brain to check my understanding of my current IP routes. Then I'm looking for any pointer toward the right approach to achieving the setup described here.

So far I have tried many rules and routes setup built along my researches, without success. I have also read a bit about flagging outgoing and incoming paquets via the mangle ip rule or route option, but the literature out there is rather hard to digest and I'd like to understand what tools I actually need before trying them further.

Any help would be very appreciated, I'll be happy to provide any detail of config or try any new config, just ask.

Answer 1

To complement the answer from Martin:

Your default route via the VPN is impairing your local VPN server because the answer to its VPN clients is sent through your remote VPN server. That way the remote client VPNs won't work anymore because the answers come from a different IP address.

You have to restrict the default route to your local network with policy routing. Maybe this quick introduction can help you set this up. This question at superuser may help too.

You would not set the default route with OpenVPN.

Then you can add the routes from your incoming VPNs to the same routing table and it should work the way you want to have it.

Answer 2

here is some insight into the routing table:

default via 172.94.109.161 dev tun0

This is your default route. All traffic which doesn't match a more specific route, will go that path. This route is being set by dhcp by default.

0.0.0.0/1 via 172.94.109.161 dev tun0
128.0.0.0/1 via 172.94.109.161 dev tun0 

These two routes come probably from openvpn - From the man page of openvpn:

def1 -- Use this flag to override the default gateway by using 0.0.0.0/1 and 128.0.0.0/1 rather than 0.0.0.0/0. This has the benefit of overriding but not wiping out the original default gateway.

So, basically, this invalidates your default gateway and replaces it by a new one - without deleting it. Without this option, the default gateway which redirects all traffic through the vpn tunnel gets overwritten as soon as the DHCP lease expires.

172.94.109.4 via 192.168.0.1 dev eth0

Whenever you start openvpn with the "redirect_gateway" option, the vpn software must add a route for the vpn gateway (the line with the remote option inside your openvpn config), which bypasses the vpn tunnel and is being sent directly with your main interface connected to your ISP - otherwise you loose connectivity immediately after the tunnel is being setup, which is what happens here most likely.

172.94.109.160/28 dev tun0 proto kernel scope link src 172.94.109.163 
192.168.0.0/24 dev eth0 proto dhcp scope link src 192.168.0.200 metric 202 

These routes are routes setup by the system - whenever you configure a network interface, such a route is created automatically - that way you can reach all neighbour hosts in the same subnet. Recommendation: never touch these routes!

From the look of the route, it looks like a wrong ip is being used... disable your "tun1" openvpn server, startup only your vpn-client instance, and compare the routes to the table above. My guess is that with this comparison, you'll find out why your setup will not work...

Additionally, you should not manually add the default route (point 3), but let openvpn do that for you.

Hope it helps!