First, let's take a look at the Packet flow in Netfilter and General Networking schematic:
At the step where OP is trying to operate: filter/INPUT it's too late to do what's intended, the occasion to do DNAT had to be done before the routing decision, in nat/PREROUTING.
So this must be earlier, and for some other cases it could have been needed to use two different tables: mangle/PREROUTING for some special test, and nat/PREROUTING for the port redirection, with a mark on the packet to communicate the intent from the mangle table to the nat table. Actually such cases would be very complex, probably involve conntrack zones in the raw table and never work with simple setups (here's such an example: IPTables redirect all UDP packets including ESTABLISHED)
Here, nat/PREROUTING will be enough. That's because only the packets starting a new flow (state NEW) are checked for OP's case, and that's exactly what already happens with the nat table: only packets in state NEW are seen (as reminded in the schematic), and every other packets will follow the NAT decision that was done. So this block:
--match conntrack --ctstate NEW becomes redundant in any nat table rule: it's always true. The limit match has no restriction and can be used in any table/chain.
In the end:
- add a filter for UDP port 5060 to not affect anything else
- no conntrack match needed here
- as a reminder,
-j ACCEPT in the nat table just means "don't do any NAT for this case"
- DECLINE-INVITE isn't really needed, but I'll leave it as used by OP
- as the DNAT target used with a port requires a protocol,
-p udp must be added again even if only 5060/UDP packets hit this rule.
- you can't easily test the result from the local system: the local system uses the path OUTPUT -> ... loopback ... -> PREROUTING -> INPUT. In the PREROUTING phase, the first incoming packet is matching the flow just created in OUTPUT before it was looped back, so it won't appear in NEW state anymore and won't see the nat/PREROUTING chain and no DNAT will be done there. DNAT can still be done immediately in OUTPUT by duplicating (and adapting adequately) these rules in nat/OUTPUT. Anyway I'm sure there are additional caveats for the local case. Do tests from remote (or from a connected container or network namespace) instead.
iptables -t nat -N RATE-LIMIT
iptables -t nat -N DECLINE-INVITE
iptables -t nat -A PREROUTING -p udp --dport 5060 -j RATE-LIMIT
iptables -t nat -A RATE-LIMIT -m limit --limit 20/sec --limit-burst 20 -j ACCEPT
iptables -t nat -A RATE-LIMIT -j DECLINE-INVITE
iptables -t nat -A DECLINE-INVITE -p udp -j DNAT --to-destination :5090