IP fragments are not reassembled due to packet loss in Linux

Question

I have been working with a device that sends heavy TCP/IP traffic.

I'm trying to capture those packets in my Linux machine (Ubuntu 22.04.2 LTS) with Python programming language.

But, most of the time the sent packets are not received correctly from the OS.

I can see some of those packets are correctly re-assembled by the OS but not most of them.

I can clearly see the from Wireshark.

Below are the unexpected behaviors:

I am mostly seeing fragmented IP protocol packets and after those, I am seeing time-to-live exceeded (fragment reassembly time exceeded).

Below is the expected behavior:

Is there a way to correct this behavior (relax the conditions that result unable to reassemble the packets) to capture all the packets?

I am never seeing this issue in Windows 10.

On the same client machine, when log out of Ubuntu and log in to Win10, I can't see dropped packets and all the transmitted chunks reassmbled and sent to corresponding port. Seems like, there are some system parameters on Linux that affects packet dropping.

I attached a Wireshark capture file below:

https://drive.google.com/file/d/1GFtlHnKF619HE02JfuSUQpIGqSxRt6Yf/view?usp=sharing

Here is the output of sudo sysctl -a | grep ipfrag

net.ipv4.ipfrag_high_thresh = 4194304
net.ipv4.ipfrag_low_thresh = 3145728
net.ipv4.ipfrag_max_dist = 64
net.ipv4.ipfrag_secret_interval = 0
net.ipv4.ipfrag_time = 30

I tried doing the following too (10 times of above settings):

sudo sysctl -w net.ipv4.ipfrag_time=300
sudo sysctl -w net.ipv4.ipfrag_high_thresh=41943040
sudo sysctl -w net.ipv4.ipfrag_low_thresh=31457280
sudo sysctl -w net.ipv4.ipfrag_max_dist=640

Still no luck.

Thanks.

Answer 1

Some fragments are getting lost for whatever reason.

This is not a reassembly issue and no amount of fiddling with timeouts is going to fix it.

What you see in Wireshark (or any pcap-based tool) is the raw communication passing to and from the NIC before it is handed over to the OS network stack. So whenever something is missing from the capture, it means the NIC never received it (or it got dropped by the hardware directly).

Your first screenshot shows a dump with at least one fragment missing (the one with offset 17760). The OS thus waits a bit for that fragment to possibly arrive (in case it got reordered in flight or retransmitted on a lower layer), and if it never arrives, you get the ICMP reassembly timeout error.

Unless there is a good reason to expect packets to be delayed for a long time by the underlying network, fiddling with timeouts is pointless. The fragments got simply lost and are never going to show up, waiting longer won't help.

This is just a reality of UDP over an unreliable transport. If the medium is significantly lossy and there's a lot of fragmentation, lots of packets will fail to reassemble.

You need to look into what is causing the packet loss in the first place. Checking the error counters for the NIC might be a good first step. If this doesn't happen on a different OS with the same hardware, chances are that either the NIC is configured differently (e.g. offloading disabled, causing the CPU to be unable to cope with the traffic, or the NIC running at a different data rate such as autonegotiated Fast Ethernet on one OS vs a forced Gigabit on the other OS).