3

Recently I'd like to set CF WARP client on my VPS, but encountered a strange problem. I’m not sure if this is the right place to ask this question, if not I’m very sorry and delete it.

Background:

I installed Cloudflare official WARP client on my VPS (Ubuntu 20.04), but once I connect to Cloudflare WARP Network, SSH disconnect (from my home to the VPS) and never connect again.

On VPS

$ warp-cli register
$ warp-cli set-mode warp
$ warp-cli connect
<<<<<<<< SSH Disconnect!

I tried to solve this problem by myself.

In WebView Console (since I can't SSH to VPS now, I have to use VPS WebView Console):

$ ip addr
1: lo ...
2: enp1s0 ...
3: CloudflareWARP ...

$ ip rule show
0: from all lookup local
32765: not from all fwmark 0x100cf lookup 65743
32766: from all lookup main 
32767: from all lookup default

Obviously all the traffic from the VPS to other sites has been taken over by WARP Network, except some packets with mark 0x100cf which be used by WARP client itself. This is why my SSH disconnected.

The standard way to solve this kind of problem (AFAIK VPN client usually disconnect SSH, e.g. WireGuard) is adding the follow rule and route:

$ ip route list table main default
default via 45.32.80.1 dev enp1s0 proto dhcp src 45.32.82.124 metric 100

$ ip rule add table 200 from 45.32.82.124
$ ip route add table 200 default via 45.32.80.1 dev enp1s0 proto dhcp src 45.32.82.124

Here 45.32.82.124 is my VPS public IP and 45.32.80.1 is the gateway.

Then

$ ip rule show
0: from all lookup local
32764: from 45.32.82.124 lookup 200
32765: not from all fwmark 0x100cf lookup 65743
32766: from all lookup main 
32767: from all lookup default

Unfortunately, the SSH still can't connect. This surprised me a lot, since in the same settings if I use WireGuard instead of WARP client, the SSH can connect.

The settings of WireGuard and WARP client are not very different.

The following is WireGuard setting:

$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq state UP group default qlen 1000
    link/ether 56:00:04:6b:06:1a brd ff:ff:ff:ff:ff:ff
    inet 107.191.58.220/24 brd 107.191.58.255 scope global dynamic enp1s0
       valid_lft 71276sec preferred_lft 71276sec
    inet6 fe80::5400:4ff:fe6b:61a/64 scope link
       valid_lft forever preferred_lft forever
3: wgclient: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN group default qlen 1000
    link/none
    inet 172.16.0.2/32 scope global wgclient
       valid_lft forever preferred_lft forever
    inet6 2606:4700:110:8cbe:b4b0:fa54:9af:e32d/128 scope global
       valid_lft forever preferred_lft forever

$ ip rule show
0:      from all lookup local
32763:  from 107.191.58.220 lookup 200
32764:  from all lookup main suppress_prefixlength 0
32765:  not from all fwmark 0xca6c lookup 51820
32766:  from all lookup main
32767:  from all lookup default

$ ip route show table local
broadcast 107.191.58.0 dev enp1s0 proto kernel scope link src 107.191.58.220
local 107.191.58.220 dev enp1s0 proto kernel scope host src 107.191.58.220
broadcast 107.191.58.255 dev enp1s0 proto kernel scope link src 107.191.58.220
broadcast 127.0.0.0 dev lo proto kernel scope link src 127.0.0.1
local 127.0.0.0/8 dev lo proto kernel scope host src 127.0.0.1
local 127.0.0.1 dev lo proto kernel scope host src 127.0.0.1
broadcast 127.255.255.255 dev lo proto kernel scope link src 127.0.0.1
local 172.16.0.2 dev wgclient proto kernel scope host src 172.16.0.2

$ ip route show table main
default via 107.191.58.1 dev enp1s0 proto dhcp src 107.191.58.220 metric 100
107.191.58.0/24 dev enp1s0 proto kernel scope link src 107.191.58.220
169.254.169.254 via 107.191.58.1 dev enp1s0 proto dhcp src 107.191.58.220 metric 100

$ ip route show table 51820
default via wgclient scope link

$ ip route show table 200
default via 107.191.58.1 dev enp1s0

The following is the WARP setting:

$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 fd01:db8:1111::3/128 scope global
       valid_lft forever preferred_lft forever
    inet6 fd01:db8:1111::2/128 scope global
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq state UP group default qlen 1000
    link/ether 56:00:04:6a:f8:84 brd ff:ff:ff:ff:ff:ff
    inet 45.32.82.124/22 brd 45.32.83.255 scope global dynamic enp1s0
       valid_lft 80142sec preferred_lft 80142sec
    inet6 fe80::5400:4ff:fe6a:f884/64 scope link
       valid_lft forever preferred_lft forever
62: CloudflareWARP: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1280 qdisc mq state UNKNOWN group default qlen 500
    link/none
    inet 172.16.0.2/32 scope global CloudflareWARP
       valid_lft forever preferred_lft forever
    inet6 2606:4700:110:85bb:76b8:7e7c:5e88:3230/128 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::7099:4d97:27a8:f1da/64 scope link stable-privacy
       valid_lft forever preferred_lft forever

$ ip rule show
0:      from all lookup local
32764:  from 45.32.82.124 lookup 200
32765:  not from all fwmark 0x100cf lookup 65743
32766:  from all lookup main
32767:  from all lookup default

$ ip route show table local
broadcast 45.32.80.0 dev enp1s0 proto kernel scope link src 45.32.82.124
local 45.32.82.124 dev enp1s0 proto kernel scope host src 45.32.82.124
broadcast 45.32.83.255 dev enp1s0 proto kernel scope link src 45.32.82.124
broadcast 127.0.0.0 dev lo proto kernel scope link src 127.0.0.1
local 127.0.0.0/8 dev lo proto kernel scope host src 127.0.0.1
local 127.0.0.1 dev lo proto kernel scope host src 127.0.0.1
broadcast 127.255.255.255 dev lo proto kernel scope link src 127.0.0.1
local 172.16.0.2 dev CloudflareWARP proto kernel scope host src 172.16.0.2

$ ip route show table main
default via 45.32.80.1 dev enp1s0 proto dhcp src 45.32.82.124 metric 100
45.32.80.0/22 dev enp1s0 proto kernel scope link src 45.32.82.124
169.254.169.254 via 45.32.80.1 dev enp1s0 proto dhcp src 45.32.82.124 metric 100

$ ip route show table 65743
0.0.0.0/5 dev CloudflareWARP proto static scope link
8.0.0.0/7 dev CloudflareWARP proto static scope link
....
#### Almost all IPs

$ ip route show table 200
default via 45.32.80.1 dev enp1s0 proto dhcp src 45.32.82.124

Note that I also tried some other settings:

  1. Add 32764: from all lookup main suppress_prefixlength 0 to the WARP setting like WireGuard does, but SSH still doesn't work

  2. When connecting SSH, on VPS I tcpdump -i enp1s0, it can log SSH messages. so the inbound traffic should be OK.

  3. Bind sshd to the VPS public IP 45.32.82.124 via /etc/ssh/sshd_config and restart sshd, but SSH still doesn't work.

  4. Tried ip route get

    $ ip route get 8.8.8.8
8.8.8.8 dev CloudflareWARP table 65743 src 172.16.0.2 uid 1001

$ ip route get 8.8.8.8 from 45.32.82.124
8.8.8.8 dev 45.32.82.124 via 45.32.80.1 dev enp1s0 table 200 uid 1001

  5. Tried ping with enp1s0

    $ ping 8.8.8.8
# This is OK, because it is using WARP Network

$ ping -I enp1s0 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data
ping: sendmesg: Operation not permitted
#### Why? 
#### It can work in WireGuard setting!

  6. Tried ping with mark

    # sudo ping -m 65743 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data
ping: sendmesg: Operation not permitted
#### Why? 
#### It can work in WireGuard setting! (65743 is the decimal of the mark 0x100cf, it should go main table, right?)

  7. Even I deleted the rule 32765: not from all fwmark 0x100cf lookup 65743, SSH still doesn't work.

  8. No change to iptables. I have checked the iptables of Wireguard and WARP settings are the same.

  9. The only way to make SSH work is using add-excluded-route for WARP client:

    warp-cli add-excluded-route <My-Home-IP>

    The effect of this command is excluding <My-Home-IP> in the table 65743, so it is trivial. But why my table 200 has no effect?

Question:

I have configured WireGuard and WARP settings exactly the same, but SSH can connect in WireGuard setting not in WARP setting. So in addition to ip rule, ip route, iptables, is there anything else that could affect routing or packet filtering, which can block my traffic, as WARP client does?

Very thanks.

Improve this question
2
  • ping: sendmesg: Operation not permitted usually points to icmp packages being blocked by the firewall. Please show the output of iptables -nvL.
    – Ned64
    Commented Apr 30, 2023 at 13:22
  • @Ned64I have pasted the output in pastebin.com/gqvb6gmW (which is almost the same as WireGuard setting).
    – chansey
    Commented Apr 30, 2023 at 18:56

2 Answers 2

Reset to default
2

I don't know Cloudflare WARP (so I cannot assess whether the following allpies here) but to answer your question: There is indeed something in Linux networking that not just affects "routing" but has higher proirity:

IPsec (or the mechanism it uses in the network layer)

ip xfrm policy list
ip xfrm state list

In general this should be helpful for understanding what's going on:

ip route get 1.2.3.4 ipproto tcp dport 22

I guess your policy routing fails for new locally-initiated connections because they do not have a from address yet.

You should add another rule:

ip rule add table 200 priority 30000 ipproto tcp dport 22
Improve this answer
7
  • "I guess your policy routing fails for new locally-initiated connections because they do not have a from address yet." yeah, this is why I tired bind sshd to the VPS public IP, but it doesn't work.
    – chansey
    Commented May 1, 2023 at 5:37
  • You said ip rule add table 200 priority 30000 ipproto tcp dport 22, do actually you mean sport? (i.e. dport was a typo?) I have tried ip rule add table 200 priority 30000 ipproto tcp sport 22, it doesn't work as well. P.s. The SSH connection is from My-Home to VPS, not VPS to other sites. i.e. VPS is the server, which WARP client installed.
    – chansey
    Commented May 1, 2023 at 5:37
  • BTW ip rule add table 200 priority 30000 ipproto tcp dport 22 doesn't work as well.
    – chansey
    Commented May 1, 2023 at 5:39
  • ip xfrm policy list and ip xfrm state list return empty.
    – chansey
    Commented May 1, 2023 at 5:40
  • Sorry, I was indeed thinking about outgoing SSH connections. Incoming connections do not have the "no from IP yet" problem and thus should be covered by your ip rule setting. Of course, you can try the sport rule in addition. I guess you didn't run ip route get. Are you sure the SSH connections are made with IPv4, not IPv6? Is there and DNAT/REDIRECT in iptables -t nat -nvL?
    – Hauke Laging
    Commented May 1, 2023 at 13:08
1

cloudflare warp uses NFT instead of iptables. when it starts, it setup set of strict firewall rules. you can add your source ip to this list via :

warp-cli add-excluded-route your-src-ip-or-network
Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .