Why is my TCP throughput much greater than UDP throughput?

Question

I haven't done anything unusual to my hardware or kernel configurations (all default settings, fresh OS install, Linux kernel 3.11 TCP/IP stack) and I'm averaging about 3.83 million messages per second through TCP while I'm only averaging 0.75 million messages per second through UDP. This seems to completely defy what I expect of the two protocols.

What's the most likely cause for the drastic difference and how can I diagnose it on Ubuntu 13.10?

#TCP RESULTS
Recv   Send    Send                          Utilization       Service Demand
Socket Socket  Message  Elapsed              Send     Recv     Send    Recv
Size   Size    Size     Time     Throughput  local    remote   local   remote
bytes  bytes   bytes    secs.    10^6bits/s  % S      % S      us/KB   us/KB

87380  65536     64    10.00      1963.43   32.96    17.09    5.500   2.852

#UDP RESULTS
Socket  Message  Elapsed      Messages                   CPU      Service
Size    Size     Time         Okay Errors   Throughput   Util     Demand
bytes   bytes    secs            #      #   10^6bits/sec % SS     us/KB

4194304      64   10.00     7491010      0      383.5     28.97    24.751
212992            10.00     1404941              71.9     25.03    21.381

For this test I have two test servers that are identical and directly connected via a 10G crossover cable. The NICs used in this case are Intel X520's with out-of-box configurations and connected onto a PCIe 3.0 x8 slot on the motherboard, which communicates with the CPU via a NUMA controller.

Answer 1

Apart from not getting detailed information about your test setup the main problem seems to be, that you use a message size of 64 byte. This is far away from the usual MTU of 1500 bytes and makes UDP highly inefficient: while TCP merges multiple sends into a single packet on the wire (except if TCP_NODELAY is set) to make efficient use of the link, each UDP message will result in a separate packet. In numbers: about 23 messages of size 64 byte will be combined into a single TCP packet of MTU size, while it will need 23 single packets for UDP for the same amount of data. Each of these packets means overhead with sending from the host, transmitting on the wire and receiving by the peer. And as seen in your case about 80% of the UDP packets get lost because your hardware is not fast enough to transmit and receive all these packets.

So what you can learn from this benchmark is:

• UDP is unreliable (80% packet loss)
• UDP is inefficient if used with packet sizes far below MTU
• TCP is highly optimized to make best use of the link

As for your expectation, that UDP should be better: did you ever wonder why all the major file transfers (ftp, http,...) are done with TCP based protocols? The benchmark shows you the reason.

So why do people use UDP at all?

• With real-time data (e.g. voice over IP) you don't care about older messages, so you don't want the sender to combine messages into larger packets to make effective use of the link. And you rather accept that a packet gets lost than to have it arrive too late.
• With high-latency links (like with satellites) the default behavior of TCP is not optimal to make effective use of the link. So some people switch to UDP in this case and re-implement the reliability layer of TCP and optimize it for high-latency links, while others tune the existing TCP stack to make better use of the link.
• "throw away" data: sometimes it is more important to send the data away and don't care about packet loss, like with log messages (syslog)
• Short interactions: with TCP you need to establish a connection an maintain a state, which costs time and resources at client and server. For short interactions (like short request and reply) this might be too much overhead. Because of this DNS is usually done with UDP but has built retries on top of UDP.