I haven't done anything unusual to my hardware or kernel configurations (all default settings, fresh OS install, Linux kernel 3.11 TCP/IP stack) and I'm averaging about 3.83 million messages per second through TCP while I'm only averaging 0.75 million messages per second through UDP. This seems to completely defy what I expect of the two protocols.
What's the most likely cause for the drastic difference and how can I diagnose it on Ubuntu 13.10?
#TCP RESULTS Recv Send Send Utilization Service Demand Socket Socket Message Elapsed Send Recv Send Recv Size Size Size Time Throughput local remote local remote bytes bytes bytes secs. 10^6bits/s % S % S us/KB us/KB 87380 65536 64 10.00 1963.43 32.96 17.09 5.500 2.852 #UDP RESULTS Socket Message Elapsed Messages CPU Service Size Size Time Okay Errors Throughput Util Demand bytes bytes secs # # 10^6bits/sec % SS us/KB 4194304 64 10.00 7491010 0 383.5 28.97 24.751 212992 10.00 1404941 71.9 25.03 21.381
For this test I have two test servers that are identical and directly connected via a 10G crossover cable. The NICs used in this case are Intel X520's with out-of-box configurations and connected onto a PCIe 3.0 x8 slot on the motherboard, which communicates with the CPU via a NUMA controller.