A variant of this problem was addressed recently in Kubernetes, so it’s worth looking at what was done there. (The variant is whether to use
iptables-nft and their IPv6 variants to drive the host’s rules.)
The approach taken in Kubernetes is to look at the number of lines output by the respective “save” commands,
iptables-nft-save (and their IPv6 variants). If the former produces ten lines or more of output, or produces more output than the latter, then it’s assumed that
iptables-legacy should be used; otherwise, that
iptables-nft should be used.
In your case, the decision tree could be as follows:
iptables isn’t installed, use
nft isn’t installed, use
iptables-save doesn’t produce any rule-defining output, use
nft list tables and
nft list ruleset don’t produce any output, use
nft list ... both produce output, and
iptables-nft, I’m not sure an automated process can decide.