1

We have a Debian Buster box (nftables 0.9.0, kernel 4.19) attached to four different network segments. Three of these segments are home to devices running Syncthing, which runs its own local discovery via broadcasts to UDP port 21027. The devices thus can't all "see" each other as the broadcasts don't cross segments; the Buster box itself does not participate in the sync cluster.

While we could solve this by running Syncthing's discovery or relay servers on the Buster box, it's been requested that we not use them (reasons around configuration and devices which roam to other sites). Hence, we're looking at a nftables-based solution; my understanding is that this isn't normally done, but to make this work, we have to:

  • Match incoming packets on UDP 21027
  • Copy those packets to the other segment interface(s) they need to be seen on
  • Change the destination IP of the new packet(s) to match the new segment's broadcast address (while preserving the source IP as the discovery protocol can rely on it)
  • Emit the new broadcasts without them getting duplicated again

Only three of the attached segments participate with devices; all are subnet masked as /24.

  • Segment A (eth0, 192.168.0.1) should not be forwarded
  • Segment B (eth1, 192.168.1.1) should be forwarded to segment A only
  • Segment C (eth2, 192.168.2.1) should be forwarded to both A and B

The closest we have to a working rule for this so far is (other DNAT/MASQ and local filtering rules omitted for brevity):

table ip mangle {
    chain repeater {
        type filter hook prerouting priority -152; policy accept;
        ip protocol tcp return
        udp dport != 21027 return
        iifname "eth1" ip saddr 192.168.2.0/24 counter ip daddr set 192.168.1.255 return
        iifname "eth0" ip saddr 192.168.2.0/24 counter ip daddr set 192.168.0.255 return
        iifname "eth0" ip saddr 192.168.1.0/24 counter ip daddr set 192.168.0.255 return
        iifname "eth2" ip saddr 192.168.2.0/24 counter dup to 192.168.0.255 device "eth0" nftrace set 1
        iifname "eth2" ip saddr 192.168.2.0/24 counter dup to 192.168.1.255 device "eth1" nftrace set 1
        iifname "eth1" ip saddr 192.168.1.0/24 counter dup to 192.168.0.255 device "eth0" nftrace set 1
    }
}

The counters show that the rules are being hit, though without the daddr set rules the broadcast address remains the same as on the originating segment. nft monitor trace shows least some packets are reaching the intended interface with the correct destination IP, but are then landing in the input hook for the box itself and are not seen by other devices on the segment.

Is the outcome we're looking for here achievable in practice, and if so, with which rules?

8
  • check nft's man page: dup to address expects a gateway address, not a broadcast. That's just a part of your issues. They are related to handling broadcasts, which involve things at the Ethernet layer, while you're trying to work at the IP layer. – A.B Aug 18 '20 at 12:31
  • I must have read that gateway address line a dozen times earlier without picking that up, ack. Switched the dup to IPs to match the interface IPs and merged the daddr set into the dup rules. Trace shows packets appear with the right IP destination on the right interfaces, but inbound rather than out. Trace also shows ether daddr ff:ff:ff:ff:ff:ff, which would seem right if it were going in the opposite direction... – T2PS Aug 18 '20 at 13:00
  • 1
    this could be solved with tc or maybe with nftables's netdev ingress (reading again it doesn't appear egress is needed, I rewrote this comment). Here's a (not really readable) comment of mine on a similar problem which handles forwarding a broadcast to an other network using tc: unix.stackexchange.com/questions/594347/… (actually tc doesn't require a bridge). I can check later if using nftables' netdev ingress is enough for the same (you can check yourself of course). – A.B Aug 18 '20 at 13:06
  • In addition for return unicast traffic, the buster box would have to be configured as a router anyway – A.B Aug 18 '20 at 13:13
  • Yes, the three segments in question are already routed/masqueraded to the fourth. We hadn't considered tc for this purpose yet, I'll take a closer look at it and nft ingress. – T2PS Aug 18 '20 at 16:36
2

It's still possible to use nftables in the netdev family (rather than ip family) for this case, since only ingress is needed (nftables still doesn't have egress available). The behaviour of dup and fwd in the ingress hook is exactly the same as tc-mirred's mirror and redirect.

I also addressed a minor detail: rewrite the Ethernet source address to the new Ethernet outgoing interface's MAC address, as would have been done for a truly routed packet, even if it works for you without this. So the interfaces' MAC addresses has to be known beforehand. I put the two required (eth0's and eth1's) in variables/macro definitions, which should be edited with the correct values.

define eth0mac = 02:0a:00:00:00:01
define eth1mac = 02:0b:00:00:00:01

table netdev statelessnat
delete table netdev statelessnat

table netdev statelessnat {
    chain b { type filter hook ingress device eth1 priority 0;
        pkttype broadcast ether type ip ip daddr 192.168.1.255 udp dport 21027 jump b-to-a
        
    }

    chain c { type filter hook ingress device eth2 priority 0;
        pkttype broadcast ether type ip ip daddr 192.168.2.255 udp dport 21027 counter jump c-to-b-a
    }

    chain b-to-a {
        ether saddr set $eth0mac ip daddr set 192.168.0.255 fwd to eth0
    }

    chain c-to-b-a {
        ether saddr set $eth1mac ip daddr set 192.168.1.255 dup to eth1 goto b-to-a
    }
}
2
  • I like this better than the tc approach. I'll try it out when the opportunity presents and make this the accepted answer once it's working. – T2PS Aug 21 '20 at 3:34
  • Tested and got clean connects for all permutations; thanks for this, accepting. – T2PS Aug 22 '20 at 20:00
2

Edit: for anyone finding this later, the accepted answer from A.B gives a purely-nft solution.

Thanks to A.B's suggestion, this is now working using tc rather than purely nftables rules:

tc qdisc add dev eth2 ingress
tc filter add dev eth2 ingress \
    protocol ip u32 \
    match ip dst 192.168.2.255 \
    match ip protocol 17 0xff \
    match ip dport 21027 0xffff \
    action nat ingress 192.168.2.255/32 192.168.0.255 \
    pipe action mirred egress mirror dev eth0 \
    pipe action nat ingress 192.168.0.255/32 192.168.1.255 \
    pipe action mirred egress redirect dev eth1

tc qdisc add dev eth1 ingress
tc filter add dev eth1 ingress \
    protocol ip u32 \
    match ip dst 192.168.1.255 \
    match ip protocol 17 0xff \
    match ip dport 21027 0xffff \
    action nat ingress 192.168.1.255/32 192.168.0.255 \
    pipe action mirred egress redirect dev eth0

My understanding of these filters is they match incoming broadcast packets for UDP port 21027, NAT them to the broadcast address for each of the other intended subnets (ingress, to change the destination IP rather than the source IP that nat egress would alter), then duplicate/redirect the NATted packets to the other interfaces' output queues.

Being a novice with tc this may not be the best way to solve the problem, but it works in terms of getting the announcement broadcasts to travel across segments (and Syncthing is happily discovering new nodes).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.