I need your help/hints/remarks on an issue I'm having. Thanks
Here is some context : I have two application(A1 & A2) handling PDUs with a max size of 1700bytes. Each app is on a different card(C1 & C2) directly connected to each other using 1Gbps Ethernet, and both app send and receive using UDP sockets. Card C2 is way more powerful the C1, A1 & A2 are alike.
Though PDUs can be up to 1700 bytes long, when each app send and receive PDUs, they are NEVER larger than 1450 bytes. So, with a typical MTU of 1500 bytes, there is a 1-to-1 mapping between PDUs and ethernet frames.
Everything works fine with various datarates except the highest one (~125Mbps FYI), where the least powerful card C1 reaches 100% CPU usage at around 90Mbps.
I started looking at how to improve global performance, by looking at the network communication. Before everything got into a working condition, I decided to start using connected UDP sockets, since I read that somehow, they were faster, by bypassing a few steps in the kernel. So I suppose they are better since I never took the time to implement its non-connected counterpart.
Remark 0 : I'm open to any response arguing why UDP connected sockets are faster
I then started to increase the MTU between C1 and C2 ==> No improvement since i still
send PDUs of 1450 bytes max over the wire.
Next step : PDU aggregation. Instead of sending PDUs 1 by 1, I aggregate them into large UDP datagrams of ~60KB and send them over the wire with an MTU of 1500 ==> No improvement (Kernel cut the big datagram at some point to end up with a ethernet frame with a typical length. No packet loss on the other end, so my mechanism must be fine.
Question 1 : Just out of curiosity, when does the kernel cut my data into smaller (MTU-like) data ? Just before sending it to IP ? Before Ethernet ? Lower ?( so the NIC is in charge of it ?)
Last step : Send the aggregated PDUs using a large MTU on both sides (3000 5000 6000 8192). There's the issue. The more I increase my MTU, the more errors I have !
Question 2 : Anyone ever experienced that ? Both card modules are expected to support ethernet jumbo frames. How could that be ? Any hint on where to look at ?
FYI, bottleneck card C1 runs kernel 2.6.23, so options like
sendmmsg are not an option.
For more information, I have two sets of 2 card. C2 retrieves 125Mbps from a traffic generator (1 PDU at a time), sends it to C1 (1 PDU at a time), which send the data flow in the air (multiple PDUs at a time). C1' catches the data right in the air (can't tell how, it's magic) (multiple PDus at a time, then send them to C2' (one at a time, but trying to send many at a time since it's not working). C2' ends the loop by sending data back to the traffic generator which compares it to the data sent earlier and prints an awful lot of details.
Thanks for your time,