nftables: duplicate broadcast packets between segments

Question

We have a Debian Buster box (nftables 0.9.0, kernel 4.19) attached to four different network segments. Three of these segments are home to devices running Syncthing, which runs its own local discovery via broadcasts to UDP port 21027. The devices thus can't all "see" each other as the broadcasts don't cross segments; the Buster box itself does not participate in the sync cluster.

While we could solve this by running Syncthing's discovery or relay servers on the Buster box, it's been requested that we not use them (reasons around configuration and devices which roam to other sites). Hence, we're looking at a nftables-based solution; my understanding is that this isn't normally done, but to make this work, we have to:

• Match incoming packets on UDP 21027
• Copy those packets to the other segment interface(s) they need to be seen on
• Change the destination IP of the new packet(s) to match the new segment's broadcast address (while preserving the source IP as the discovery protocol can rely on it)
• Emit the new broadcasts without them getting duplicated again

Only three of the attached segments participate with devices; all are subnet masked as /24.

• Segment A (eth0, 192.168.0.1) should not be forwarded
• Segment B (eth1, 192.168.1.1) should be forwarded to segment A only
• Segment C (eth2, 192.168.2.1) should be forwarded to both A and B

The closest we have to a working rule for this so far is (other DNAT/MASQ and local filtering rules omitted for brevity):

table ip mangle {
    chain repeater {
        type filter hook prerouting priority -152; policy accept;
        ip protocol tcp return
        udp dport != 21027 return
        iifname "eth1" ip saddr 192.168.2.0/24 counter ip daddr set 192.168.1.255 return
        iifname "eth0" ip saddr 192.168.2.0/24 counter ip daddr set 192.168.0.255 return
        iifname "eth0" ip saddr 192.168.1.0/24 counter ip daddr set 192.168.0.255 return
        iifname "eth2" ip saddr 192.168.2.0/24 counter dup to 192.168.0.255 device "eth0" nftrace set 1
        iifname "eth2" ip saddr 192.168.2.0/24 counter dup to 192.168.1.255 device "eth1" nftrace set 1
        iifname "eth1" ip saddr 192.168.1.0/24 counter dup to 192.168.0.255 device "eth0" nftrace set 1
    }
}

The counters show that the rules are being hit, though without the daddr set rules the broadcast address remains the same as on the originating segment. nft monitor trace shows least some packets are reaching the intended interface with the correct destination IP, but are then landing in the input hook for the box itself and are not seen by other devices on the segment.

Is the outcome we're looking for here achievable in practice, and if so, with which rules?

Answer 1

It's still possible to use nftables in the netdev family (rather than ip family) for this case, since only ingress is needed (nftables still doesn't have egress available). The behaviour of dup and fwd in the ingress hook is exactly the same as tc-mirred's mirror and redirect.

I also addressed a minor detail: rewrite the Ethernet source address to the new Ethernet outgoing interface's MAC address, as would have been done for a truly routed packet, even if it works for you without this. So the interfaces' MAC addresses has to be known beforehand. I put the two required (eth0's and eth1's) in variables/macro definitions, which should be edited with the correct values.

define eth0mac = 02:0a:00:00:00:01
define eth1mac = 02:0b:00:00:00:01

table netdev statelessnat
delete table netdev statelessnat

table netdev statelessnat {
    chain b { type filter hook ingress device eth1 priority 0;
        pkttype broadcast ether type ip ip daddr 192.168.1.255 udp dport 21027 jump b-to-a
        
    }

    chain c { type filter hook ingress device eth2 priority 0;
        pkttype broadcast ether type ip ip daddr 192.168.2.255 udp dport 21027 counter jump c-to-b-a
    }

    chain b-to-a {
        ether saddr set $eth0mac ip daddr set 192.168.0.255 fwd to eth0
    }

    chain c-to-b-a {
        ether saddr set $eth1mac ip daddr set 192.168.1.255 dup to eth1 goto b-to-a
    }
}

Answer 2

Edit: for anyone finding this later, the accepted answer from A.B gives a purely-nft solution.

Thanks to A.B's suggestion, this is now working using tc rather than purely nftables rules:

tc qdisc add dev eth2 ingress
tc filter add dev eth2 ingress \
    protocol ip u32 \
    match ip dst 192.168.2.255 \
    match ip protocol 17 0xff \
    match ip dport 21027 0xffff \
    action nat ingress 192.168.2.255/32 192.168.0.255 \
    pipe action mirred egress mirror dev eth0 \
    pipe action nat ingress 192.168.0.255/32 192.168.1.255 \
    pipe action mirred egress redirect dev eth1

tc qdisc add dev eth1 ingress
tc filter add dev eth1 ingress \
    protocol ip u32 \
    match ip dst 192.168.1.255 \
    match ip protocol 17 0xff \
    match ip dport 21027 0xffff \
    action nat ingress 192.168.1.255/32 192.168.0.255 \
    pipe action mirred egress redirect dev eth0

My understanding of these filters is they match incoming broadcast packets for UDP port 21027, NAT them to the broadcast address for each of the other intended subnets (ingress, to change the destination IP rather than the source IP that nat egress would alter), then duplicate/redirect the NATted packets to the other interfaces' output queues.

Being a novice with tc this may not be the best way to solve the problem, but it works in terms of getting the announcement broadcasts to travel across segments (and Syncthing is happily discovering new nodes).