Unwanted defragmentation of forwarded ipv4 packets

Question

I want to handle ip fragments in user-space, and I am using the iptables NF_QUEUE to direct packets to user-space.

The problem is that IPv4 packets are always re-assembled and delivered as one packet rather than individual fragments. For IPv6, fragments are delivered as they should.

I thought that the conntracker might be causing it and disabled it in the raw iptables table, but it turns out that the packet is already re-assembled when it reaches the raw table:

# iptables -t raw -nvL
Chain PREROUTING (policy ACCEPT 58 packets, 62981 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    1 30028 CT         all  --  *      *       0.0.0.0/0            10.0.0.0/24          NOTRACK

This is when sending a 30000 byte UDP packet over IPv4. The corresponding for IPv6:

# ip6tables -t raw -nvL
Chain PREROUTING (policy ACCEPT 46 packets, 62304 bytes)
 pkts bytes target     prot opt in     out     source               destination         
   21 31016 CT         all      *      *       ::/0                 1000::               NOTRACK

This is in a virtual environment kvm/qemu with virtio network devices, mtu=1500. Some HW offload does not seem to cause this, since I can see all IPv4 fragments with tcpdump -ni eth2 host 10.0.0.0.

So my question is what in the Linux kernel can force IPv4 packets to be re-assembled before the raw/PREROUTING netfilter chain?

I suspect "ingress/qdisc" as it is in between AF_PACKET (tcpdump) and the raw/PREROUTING chain, but I can't find the problem.

Packet flow: https://upload.wikimedia.org/wikipedia/commons/3/37/Netfilter-packet-flow.svg

Answer 1

Whenever conntrack is in use, mainly for:

• stateful firewalling (-m conntrack ...)
• NAT (-t nat ...)

An additional hidden facility gets loaded, provided by the kernel modules nf_defrag_ipv4 and nf_defrag_ipv6. This facility hooks into network prerouting at priority -400: that's before iptables' raw table which hooks at priority -300. After a packet(s) traverses nf_defrag_ipv[46] no fragment ever exists: packets were reassembled early. The goal is that the various protocol inspectors in Netfilter and iptables can get all of the packet contents, including for example UDP destination port: this information will be present only in the first fragment.

So to avoid this, the -j NOTRACK (obsoleted by -j CT --notrack) in the raw table is not enough.

One can:

• never use conntrack directly (stateful rules) or indirectly (NAT),

• or create a new network namespace

  to handle directly traffic (most likely using a stolen physical interface, or a macvlan interface, or being bridged but not routed by the host) and making sure that no stateful rule happens in this namespace. the defragmentation facility doesn't hook in the network namespace as long as nothing forces it to do so (and probably also with a recent enough kernel)

• or have a chain that hooks before priority -400. This is actually possible in recent enough kernels:

  • for iptables-legacy since kernel (probably) >= 4.16

    # modinfo -p iptable_raw
    raw_before_defrag:Enable raw table before defrag (bool)
    

    flush raw table, unload module and reload it with (and adjust /etc/modprobe.d/):

    modprobe iptable_raw raw_before_defrag=1
    

  • for nftables

    Just create chains with priority lower than -400, eg:

    nft add table ip handlefrag
    nft add chain ip handlefrag predefrag '{ type filter hook prerouting priority -450; policy accept; }'
    nft add rule ip handlefrag predefrag ip 'frag-off & 0x3fff != 0' notrack 
    

    (to deal only with following fragments, not the first, replace 0x3fff with 0x1fff)

    For IPv6 the method is different, because the fragment header might not be the next header. But nft provides an easy expression in its man: exthdr frag exists to detect packets that are fragments.

  • nothing exists for the iptables-nft API (which is default on many distributions like Debian): it doesn't use the module iptable_raw and doesn't have an option to create the actual nftables chain at priority -450.

    So if your command's output looks like this:

    # iptables -V
    iptables v1.8.7 (nf_tables)
    

    you can't use this alone along conntrack. You must revert to iptables-legacy or switch to nft, or...

  • what can yet still be done, is to mix nftables (rules above) to tag the packets as notrack before they are defragmented, and continue with iptables to handle the remaining part. There's no issue in using nftables and iptables at the same time, as long one understands the order of operations as seen in the schematic of Netfilter linked by OP.

Answer 2

iptables has (for IPv4) the -f flag. Adding it should work. Check also with the INPUT chain.

[!] -f, --fragment

This means that the rule only refers to second and further IPv4 fragments of fragmented packets. Since there is no way to tell the source or destination ports of such a packet (or ICMP type), such a packet will not match any rules which specify them. When the "!" argument precedes the "-f" flag, the rule will only match head fragments, or unfragmented packets. This option is IPv4 specific, it is not available in ip6tables.