I haven't done anything unusual to my hardware or kernel configurations (all default settings, fresh OS install, Linux kernel 3.11 TCP/IP stack) and I'm averaging about 3.83 million messages per second through TCP while I'm only averaging 0.75 million messages per second through UDP. This seems to completely defy what I expect of the two protocols.

What's the most likely cause for the drastic difference and how can I diagnose it on Ubuntu 13.10?

Recv   Send    Send                          Utilization       Service Demand
Socket Socket  Message  Elapsed              Send     Recv     Send    Recv
Size   Size    Size     Time     Throughput  local    remote   local   remote
bytes  bytes   bytes    secs.    10^6bits/s  % S      % S      us/KB   us/KB

87380  65536     64    10.00      1963.43   32.96    17.09    5.500   2.852

Socket  Message  Elapsed      Messages                   CPU      Service
Size    Size     Time         Okay Errors   Throughput   Util     Demand
bytes   bytes    secs            #      #   10^6bits/sec % SS     us/KB

4194304      64   10.00     7491010      0      383.5     28.97    24.751
212992            10.00     1404941              71.9     25.03    21.381

For this test I have two test servers that are identical and directly connected via a 10G crossover cable. The NICs used in this case are Intel X520's with out-of-box configurations and connected onto a PCIe 3.0 x8 slot on the motherboard, which communicates with the CPU via a NUMA controller.

  • How did you the benchmarks? Against what you sent those packages?
    – Braiam
    Mar 30, 2014 at 18:39
  • I used netperf for the benchmarks, UDP_STREAM and TCP_STREAM tests, fixed to same CPU, and 64 byte message sizes.
    – elleciel
    Mar 30, 2014 at 18:40
  • 1
    That doesn't answer @Braiam's question. Network topology is and detailed testing method is important, here. Mar 30, 2014 at 18:57
  • 1
    @PavelŠimerda Sorry, I thought he was only asking for the testing methodology only. Regarding the network topology, the two test servers are identical and directly connected via a 10G crossover cable. The NICs used in this case are Intel X520's with out-of-box configurations and connected onto a PCIe 3.0 x8 slot on the motherboard, which communicates with the CPU via a NUMA controller. Does this answer your question?
    – elleciel
    Mar 30, 2014 at 19:39
  • 1
    Yes, @elleciel, it definitely answers my question. Although in this case I don't have the expertise to give you the answer for directly connected machines. I see you amended the question itself, which is great. Will up the question as now I'm also interested. Mar 30, 2014 at 21:26

1 Answer 1


Apart from not getting detailed information about your test setup the main problem seems to be, that you use a message size of 64 byte. This is far away from the usual MTU of 1500 bytes and makes UDP highly inefficient: while TCP merges multiple sends into a single packet on the wire (except if TCP_NODELAY is set) to make efficient use of the link, each UDP message will result in a separate packet. In numbers: about 23 messages of size 64 byte will be combined into a single TCP packet of MTU size, while it will need 23 single packets for UDP for the same amount of data. Each of these packets means overhead with sending from the host, transmitting on the wire and receiving by the peer. And as seen in your case about 80% of the UDP packets get lost because your hardware is not fast enough to transmit and receive all these packets.

So what you can learn from this benchmark is:

  • UDP is unreliable (80% packet loss)
  • UDP is inefficient if used with packet sizes far below MTU
  • TCP is highly optimized to make best use of the link

As for your expectation, that UDP should be better: did you ever wonder why all the major file transfers (ftp, http,...) are done with TCP based protocols? The benchmark shows you the reason.

So why do people use UDP at all?

  • With real-time data (e.g. voice over IP) you don't care about older messages, so you don't want the sender to combine messages into larger packets to make effective use of the link. And you rather accept that a packet gets lost than to have it arrive too late.
  • With high-latency links (like with satellites) the default behavior of TCP is not optimal to make effective use of the link. So some people switch to UDP in this case and re-implement the reliability layer of TCP and optimize it for high-latency links, while others tune the existing TCP stack to make better use of the link.
  • "throw away" data: sometimes it is more important to send the data away and don't care about packet loss, like with log messages (syslog)
  • Short interactions: with TCP you need to establish a connection an maintain a state, which costs time and resources at client and server. For short interactions (like short request and reply) this might be too much overhead. Because of this DNS is usually done with UDP but has built retries on top of UDP.
  • 2
    You should also have a look at your 80% packet loss with UDP. It looks like your hardware is not fast enough to process the packets in the same speed they get send. While TCP adapts to this kind of packet loss with slowing down, UDP will just send at the same speed and continue to loose packets. But at the end it is not relevant how fast you can send, but what you receive. Mar 30, 2014 at 19:56
  • 1
    Something else that might be a factor is TCP acceleration/offloading to the network card (if it supports it). Mar 30, 2014 at 20:03
  • 1
    Packet sending might be more efficient than receiving, especially if the last one is interrupt driven. Mar 30, 2014 at 20:11
  • 1
    people also use UDP for an embedded device to broadcast the data it is collecting over a wire and not bother with the connection setup Mar 30, 2014 at 22:31
  • 3
    You are most likely IO bound by the PCI express bus. The network cards will have TCP segment offloading enabled, most likely. This means that TCP transfers will be sent to the card asone big block, then the card slices and dices them into packets and puts them on the wire. There is no equivalent for UDP, so the result is one PCIe transaction (and all associated overheads) for each packet. Mar 31, 2014 at 2:54

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .