3

I need your help/hints/remarks on an issue I'm having. Thanks

Here is some context : I have two application(A1 & A2) handling PDUs with a max size of 1700bytes. Each app is on a different card(C1 & C2) directly connected to each other using 1Gbps Ethernet, and both app send and receive using UDP sockets. Card C2 is way more powerful the C1, A1 & A2 are alike.

Though PDUs can be up to 1700 bytes long, when each app send and receive PDUs, they are NEVER larger than 1450 bytes. So, with a typical MTU of 1500 bytes, there is a 1-to-1 mapping between PDUs and ethernet frames.

Everything works fine with various datarates except the highest one (~125Mbps FYI), where the least powerful card C1 reaches 100% CPU usage at around 90Mbps.

I started looking at how to improve global performance, by looking at the network communication. Before everything got into a working condition, I decided to start using connected UDP sockets, since I read that somehow, they were faster, by bypassing a few steps in the kernel. So I suppose they are better since I never took the time to implement its non-connected counterpart.

Remark 0 : I'm open to any response arguing why UDP connected sockets are faster

I then started to increase the MTU between C1 and C2 ==> No improvement since i still send PDUs of 1450 bytes max over the wire.

Next step : PDU aggregation. Instead of sending PDUs 1 by 1, I aggregate them into large UDP datagrams of ~60KB and send them over the wire with an MTU of 1500 ==> No improvement (Kernel cut the big datagram at some point to end up with a ethernet frame with a typical length. No packet loss on the other end, so my mechanism must be fine.

Question 1 : Just out of curiosity, when does the kernel cut my data into smaller (MTU-like) data ? Just before sending it to IP ? Before Ethernet ? Lower ?( so the NIC is in charge of it ?)

Last step : Send the aggregated PDUs using a large MTU on both sides (3000 5000 6000 8192). There's the issue. The more I increase my MTU, the more errors I have !

Question 2 : Anyone ever experienced that ? Both card modules are expected to support ethernet jumbo frames. How could that be ? Any hint on where to look at ?

FYI, bottleneck card C1 runs kernel 2.6.23, so options like sendmmsg are not an option.

For more information, I have two sets of 2 card. C2 retrieves 125Mbps from a traffic generator (1 PDU at a time), sends it to C1 (1 PDU at a time), which send the data flow in the air (multiple PDUs at a time). C1' catches the data right in the air (can't tell how, it's magic) (multiple PDus at a time, then send them to C2' (one at a time, but trying to send many at a time since it's not working). C2' ends the loop by sending data back to the traffic generator which compares it to the data sent earlier and prints an awful lot of details.

Thanks for your time,

SOKS

  • FYI, the term for this is "IP fragmentation". It happens in the IP layer, not UDP, whenever an IP datagram is larger than the MTU of the link. – Barmar Apr 5 '17 at 20:11
  • Thanks ! I noticed that the path MTU discovery is enabled. I guess IP fragments my datagram in chunks of PATH_MTU, which SHOULD be equal to the MTU, since both cards have the same MTU set and are directly connected to each other. Regarding the TX loss, ifconfig on the card aggregating PDUs reports TX errors (and the exact same amount of drops)j, which is EXACTLY the same amount of loss packets reported by my traffic generator. Any idea why I'm losing around 1% of the TX packets ? – SOKS Apr 6 '17 at 8:08
  • Packets can be dropped if the link can't keep up, or you run out of kernel packet buffers. – Barmar Apr 6 '17 at 17:05
  • This is why higher-layer protocols generally try to avoid fragmentation -- if a single fragment is lost, the entire datagram has to be resent. Path MTU discovery allows upper protocols to match their PDUs to the path MTU. – Barmar Apr 6 '17 at 17:08
  • Thing is : with MTU = 1500bytes , I can handle 80Mbps without any losses. with MTU >= 1500bytes, I get errors even before reaching 80Mbps. I send less (but larger packets). Shouldn't it be (at least a little bit) less consuming ? Btw, thanks for your answers – SOKS Apr 7 '17 at 10:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.