Outgoing docker packets not respecting routing rules

Question

I have a private docker host (A) connected via a Wireguard tunnel to a public "router" in a cloud environment (B).

Wireguard:

(Docker Container)
eth0 172.17.0.2
|
docker0 172.17.0.1
(A) eth0 192.168.1.2 --- Internet (can't accept conns)
wg0 10.0.16.6
|
wg0 10.0.16.5
(B) 
eth0 public IP Q
|
USER

The idea is to have traffic first go to B on it's public IP and then forward the request to B and forward the responses back.

I have followed this guide, specifically the Policy Routing section.

The wireguard configuration on A is as follows:

[Interface]
Address = 10.0.16.6/30
Table = 123

PreUp = ip rule add from 10.0.16.6 table 123 priority 456
PostDown = ip rule del from 10.0.16.6 table 123 priority 456

[Peer]
AllowedIPs = 0.0.0.0/0
Endpoint = Q:51820
PersistentKeepalive = 30

The packet with source 10.0.16.6 and destination of the original sender should be routed over the wg0 interface because of this rule:

0:  from all lookup local
456:    from 10.0.16.6 lookup 123 ### This one
32766:  from all lookup main
32767:  from all lookup default

So on A, a packet from B enters from the wireguard interface (with destination address 10.0.16.6) and gets routed to the docker http server correctly.

Then as the response packet comes back out from the docker container to A, it has the container's source address. This means that the routing rule above doesn't get respected like it would if the http server was running on the host A.

The packet ends up being NATed after the routing and then sent off on A eth0 instead of A wg0.

I would like the packets to get re-routed after the NAT stage.

Any regular traffic leaving the docker container should not go over the wg0 interface, only traffic originating from the wg0 interface as the referenced guide suggests.

If any further information is required, I am happy to provide e.g. iptables -L etc

Answer 1

If you wanted all the traffic from the Docker container to go out the Docker host's WireGuard connection, you could use the policy routing technique you've outlined -- but using the Docker container's IP address in the policy routing rule:

ip rule add from 172.17.0.2 table 123 priority 456

The Docker host's IP address won't work for this rule to route traffic forwarded from the container, however, because SNAT is applied after the host has already made its routing decision.

So if you need only return traffic that originated through the WireGuard connection to be routed back out the WireGuard connection, you'll need to use connection marking on the Docker host. The core of this technique is three steps:

1. Set a custom connection mark on new connections incoming through the WireGuard interface:

    iptables -t mangle -A PREROUTING -i wg0 -m state --state NEW -j CONNMARK --set-mark 1
   

2. On outgoing connections, copy the connection mark to the packet mark:

    iptables -t mangle -A PREROUTING ! -i wg0 -m connmark --mark 1 -j MARK --set-mark 1
   

3. Use the packet mark in your policy routing rules to route packets back out the WireGuard interface:

    ip rule add fwmark 1 table 123 priority 456
   

See the "Connmark on the Intermediate Server" section of the companion article to the guide you linked for more notes and caveats: WireGuard Port Forwarding From the Internet to Other Networks.