2

I use WireGuard as a secure communication channel between two servers in different DCs to hide the existence of the end server (server B). I use nftables as a firewall management tool.

From public server A, the traffic is forwarded keeping the IP address (necessary for the application).

The destination server B receives the packets and the application processes them, but eventually, the server tries to return the packet to the original IP (the sender IP of the original packet) and this becomes a problem.

Masquerading the original IP looks like a simple solution, but it is necessary to keep the original IP already on server B to route these packets back to the WireGuard tunnel.

The tcpdump of the server B:

1:02:36.675958 wg0   In  IP 1.2.3.4.54617 > 10.0.0.2.21: Flags [S], seq 1265491449, win 64240, options [mss 1452,nop,wscale 8,nop,nop,sackOK], length 0
1:02:36.675980 docker0 Out IP 1.2.3.4.54617 > 172.16.0.2.21: Flags [S], seq 1265491449, win 64240, options [mss 1452,nop,wscale 8,nop,nop,sackOK], length 0
1:02:36.676030 docker0 In  IP 172.16.0.2.21 > 1.2.3.4.54617: Flags [S.], seq 1815055360, ack 1265491450, win 64240, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
1:02:36.676033 enp41s0 Out IP 10.0.0.2.21 > 1.2.3.4.54617: Flags [S.], seq 1815055360, ack 1265491450, win 64240, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
  • 1.2.3.4 - original IP
  • 10.0.0.2 - IP wireguard on B server
  • 172.16.0.2 - docker network, let's pretend this is an application (everything works there correctly)

Unfortunately, I have not come up with a solution to this problem, so I am asking for your help. Is it still possible? If yes, by what means?

Update

I decided to use HAProxy, but I don't think that's a very high-performance solution. So I'm still in need of possible solutions to this problem.

Use systemd-networkd to configure the WireGuard tunnel:

# sudo cat /etc/systemd/network/99-wg0.netdev

[NetDev]
Name=wg0
Kind=wireguard
Description=WireGuard tunnel wg0

[WireGuard]
ListenPort=51820
PrivateKey=[key]

[WireGuardPeer]
PublicKey=[key]
PresharedKey=[key]
AllowedIPs=0.0.0.0/0
Endpoint=[server A]:51820

# sudo cat /etc/systemd/network/99-wg0.network

[Match]
Name=wg0

[Network]
Address=10.0.0.2/24
Address=fdc9:281f:04d7:9ee9::2/64

# sudo ip rule:

0:      from all lookup local
32766:  from all lookup main
32767:  from all lookup default

nftables configurations:

# sudo nft list ruleset         
                                                                                                                                                               [0]
table inet filter {
        chain allow {
                ct state invalid drop comment "early drop of invalid connections"
                ct state { established, related } accept comment "allow tracked connections"

                ip protocol icmp accept comment "allow icmp"
                meta l4proto ipv6-icmp accept comment "allow icmp v6"

                icmp type echo-request limit rate over 10/second burst 4 packets drop comment "No ping floods"
                icmpv6 type echo-request limit rate over 10/second burst 4 packets drop comment "No ping floods"
        }

        chain wireguard {
                tcp dport 21 accept
        }

        chain input {
                type filter hook input priority filter; policy drop;

                iif "lo" accept comment "allow from loopback"

                tcp dport 22 ct state new limit rate 15/minute accept comment "Avoid brute force on SSH"
                tcp dport 22 accept comment "allow sshd"

                ip6 saddr [server A] udp dport 51820 accept comment "Accept wireguard connection from proxy1.vps-da4c9ada.ovh.zolotomc.ru"

                jump allow comment "allowed traffic for input"

                meta pkttype host limit rate 5/second counter packets 1047 bytes 43403 reject with icmpx admin-prohibited
                reject with icmpx host-unreachable
        }

        chain forward {
                type filter hook forward priority filter; policy drop;

                iif "docker0" accept comment "allow outgoing traffic from docker"

                jump allow comment "allowed traffic for forward"

                iif "wg0" jump wireguard comment "Wireguard chain"

                reject with icmpx host-unreachable
        }

        chain output {
                type filter hook output priority filter; policy accept;
        }
}

table inet nat {
        chain prerouting {
                type nat hook prerouting priority dstnat; policy accept;
                iif "wg0" jump wireguard comment "Wireguard chain"
        }

        chain wireguard {
                tcp dport 21 dnat ip to 172.16.0.2
                tcp dport 21 dnat ip6 to fe80::a8d7:f6ff:fe0b:4774
        }

        chain input {
                type nat hook input priority 100; policy accept;
        }

        chain output {
                type nat hook output priority -100; policy accept;
        }

        chain postrouting {
                type nat hook postrouting priority srcnat; policy accept;
                iif "docker0" oif != "docker0" masquerade
        }
}

I tried using Setting packet metainformation, but it reassigns the port to a new random port. And yet, I'm starting to think that it's impossible or too complicated based on just nftables rules.

4
  • Can you provide your wg-quick's configuration on server B, as well as the result of ip rule?
    – A.B
    Sep 21, 2022 at 19:03
  • @A.B, see the update on the question. Sep 21, 2022 at 20:09
  • I'm surprised that There is neither Table=off nor Table=auto (which is the default and adds an ip rule entry). I'll assume systemd-networkd doesn't have the same behavior as wg-quick when configuring the WireGuard interface.
    – A.B
    Sep 21, 2022 at 20:38
  • @A.B, accessing server B on server A via WireGuard already forwards the access from wg0 directly to the server. Server A -[ WireGuard ]-> Server B -[ Netfilter ]-> Local B. If you needed it. I find systemd-networkd easier to configure Sep 21, 2022 at 21:35

1 Answer 1

3

The required information to reply correctly: "From which interface did the initial packet come from?" is lost when a different packet is seen in reply coming from a network interface. A way to memorize it and reuse it in replies is needed. Here it is: conntrack which memorizes a list of all currently tracked connections. It can also store, per flow a mark which is then called connmark. One can give a meaning to a value.

This allows to apply policy routing to a whole flow rather than just to an individual packet.

The blog To Linux and beyond ! explains it there: Netfilter Connmark.

The idea is to memorize the information when a flow is using the WireGuard interface so that reply packets are associated to the same flow, and routed back through the WireGuard interface instead of the usual route they should have taken. This can be handled by an independant nftables table and associated routing tables/rules to alter the normal fate of the packet. Using arbitrary mark value 0xf00 with the meaning "this flow came from wg0" and arbitrary routing table 1000 to select wg0 for replies.

Prepare the adequate routing table and routing rules to override the normal routes for reply packets:

ip route add default dev wg0 table 1000
ip rule add fwmark 0xf00 lookup 1000

If IPv6 is also used (but OP's configuration lacks IPv6 in AllowedIPs) then also:

ip -6 route add default dev wg0 table 1000
ip -6 rule add fwmark 0xf00 lookup 1000

example using systemd-networkd:

[Match]
Name=wg0

[Network]
Address=10.0.0.2/24
Address=fdc9:281f:04d7:9ee9::2/64

[Route]
Gateway=0.0.0.0
Table=1000

[RoutingPolicyRule]
Table=1000
FirewallMark=0xf00

And add the table below:

replywg0.nft:

table inet replywg0         # for idempotency
delete table inet replywg0  # for idempotency

table inet replywg0 {
    chain prerouting {
        type filter hook prerouting priority -150; policy accept;
        iif wg0 ct mark set 0xf00
        iif != wg0 ct mark 0xf00 meta mark set 0xf00
    }

    chain output {
        type route hook output priority -150; policy accept;
        ct mark 0xf00 meta mark set 0xf00
    }
}

load with:

nft -f replywg0.nft

Notes and caveats:

  • the packet mark, the only one which affects routing, is not set when coming from wg0, only the connmark is set: else it would select the single route defined in table 1000 and would route back the packet from where it came instead of proceeding to the local system or to Docker containers. If routing table 1000 included all the (unknown to me) relevant additional routes, there would be no need for this special handling.

  • the output chain is completely optional if that's only for Docker traffic. It would be useful only for local reply traffic: traffic received from wg0 that didn't go to Docker but to the local system. Note the use of type route instead of type filter here, so a new route lookup can still happen. Anyway some UDP services won't reply properly here: the IP source address would be the wrong one (the one set on enp41s0) and some additional imperfect NAT band-aid would be needed (for example masquerade in table inet nat output).

  • Be careful if other needs for marks are put in place later (even WireGuard itself has an option that can set marks): they will likely clash if not handled together properly.

  • FTP (which appears in OP's settings) is special: it uses additional connections for data and can require an ALG (usually in passive mode for the server case). As long as it's not encrypted, on Linux it can be handled with the kernel modules nf_conntrack_ftp+nf_nat_ftp and adequate settings, which are described for example there: Secure use of iptables and connection tracking helpers and have (subtly different) equivalent nftables settings. As RELATED traffic (the data traffic) inherits the connmark, the settings described in that blog would even work along the settings in this answer. The toggle nf_conntrack_helper is being completely removed from kernels >= 6.0, so this becomes a required setting.

  • I'm assuming that rp_filter has not the value 1 because else probably nothing would have worked through wg0 in the first place. If it was set to 1, additional settings would be required in several places.

6
  • Thank you very much for your time! I am very grateful to you! I too thought in this direction and understood how it should work, but didn't know about how to implement it. I don't know much about ip route and their routing tables. I need to read more documentation to fully understand this implementation. I will try this method soon and give you feedback. You are the best! Sep 22, 2022 at 18:42
  • ct mark 0xf00 meta mark set 0xf00 in output chain corrupts the output packets for the end user, but it seems to be possible to clean up on the proxy side. Dec 14, 2022 at 23:18
  • A mark is not part of the payload sent on the wire. So I have trouble understanding how this would corrupt the packet. It can be misrouted (and using the mark is part of the attempt to route it correctly), but not corrupted. Can you give more information? Also is it working or not? Because if you accepted the answer that would mean it's working.
    – A.B
    Dec 15, 2022 at 14:16
  • MTU issues can still happen with a tunnel if ICMP packets are dropped. If that were the case some bandaid can be applied for this.
    – A.B
    Dec 15, 2022 at 14:19
  • Yes, it does work, but some end-users are resetting the connection with the ct mark 0xf00 meta mark set 0xf00 nftables rule. Personally, it worked for me with this rule too. I can assume that the user's ISPs use this mark for their own needs. Dec 16, 2022 at 19:14

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .