Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I haven't done anything unusual to my hardware or kernel configurations (all default settings, fresh OS install, Linux kernel 3.11 TCP/IP stack) and I'm averaging about 3.83 million messages per second through TCP while I'm only averaging 0.75 million messages per second through UDP. This seems to completely defy what I expect of the two protocols.

What's the most likely cause for the drastic difference and how can I diagnose it on Ubuntu 13.10?

#TCP RESULTS
Recv   Send    Send                          Utilization       Service Demand
Socket Socket  Message  Elapsed              Send     Recv     Send    Recv
Size   Size    Size     Time     Throughput  local    remote   local   remote
bytes  bytes   bytes    secs.    10^6bits/s  % S      % S      us/KB   us/KB

87380  65536     64    10.00      1963.43   32.96    17.09    5.500   2.852

#UDP RESULTS
Socket  Message  Elapsed      Messages                   CPU      Service
Size    Size     Time         Okay Errors   Throughput   Util     Demand
bytes   bytes    secs            #      #   10^6bits/sec % SS     us/KB

4194304      64   10.00     7491010      0      383.5     28.97    24.751
212992            10.00     1404941              71.9     25.03    21.381

For this test I have two test servers that are identical and directly connected via a 10G crossover cable. The NICs used in this case are Intel X520's with out-of-box configurations and connected onto a PCIe 3.0 x8 slot on the motherboard, which communicates with the CPU via a NUMA controller.

share|improve this question
    
How did you the benchmarks? Against what you sent those packages? –  Braiam Mar 30 at 18:39
    
I used netperf for the benchmarks, UDP_STREAM and TCP_STREAM tests, fixed to same CPU, and 64 byte message sizes. –  elleciel Mar 30 at 18:40
1  
That doesn't answer @Braiam's question. Network topology is and detailed testing method is important, here. –  Pavel Šimerda Mar 30 at 18:57
1  
@PavelŠimerda Sorry, I thought he was only asking for the testing methodology only. Regarding the network topology, the two test servers are identical and directly connected via a 10G crossover cable. The NICs used in this case are Intel X520's with out-of-box configurations and connected onto a PCIe 3.0 x8 slot on the motherboard, which communicates with the CPU via a NUMA controller. Does this answer your question? –  elleciel Mar 30 at 19:39
1  
Yes, @elleciel, it definitely answers my question. Although in this case I don't have the expertise to give you the answer for directly connected machines. I see you amended the question itself, which is great. Will up the question as now I'm also interested. –  Pavel Šimerda Mar 30 at 21:26

1 Answer 1

up vote 23 down vote accepted

Apart from not getting detailed information about your test setup the main problem seems to be, that you use a message size of 64 byte. This is far away from the usual MTU of 1500 bytes and makes UDP highly inefficient: while TCP merges multiple sends into a single packet on the wire (except if TCP_NODELAY is set) to make efficient use of the link, each UDP message will result in a separate packet. In numbers: about 23 messages of size 64 byte will be combined into a single TCP packet of MTU size, while it will need 23 single packets for UDP for the same amount of data. Each of these packets means overhead with sending from the host, transmitting on the wire and receiving by the peer. And as seen in your case about 80% of the UDP packets get lost because your hardware is not fast enough to transmit and receive all these packets.

So what you can learn from this benchmark is:

  • UDP is unreliable (80% packet loss)
  • UDP is inefficient if used with packet sizes far below MTU
  • TCP is highly optimized to make best use of the link

As for your expectation, that UDP should be better: did you ever wonder why all the major file transfers (ftp, http,...) are done with TCP based protocols? The benchmark shows you the reason.

So why do people use UDP at all?

  • With real-time data (e.g. voice over IP) you don't care about older messages, so you don't want the sender to combine messages into larger packets to make effective use of the link. And you rather accept that a packet gets lost than to have it arrive too late.
  • With high-latency links (like with satellites) the default behavior of TCP is not optimal to make effective use of the link. So some people switch to UDP in this case and re-implement the reliability layer of TCP and optimize it for high-latency links, while others tune the existing TCP stack to make better use of the link.
  • "throw away" data: sometimes it is more important to send the data away and don't care about packet loss, like with log messages (syslog)
  • Short interactions: with TCP you need to establish a connection an maintain a state, which costs time and resources at client and server. For short interactions (like short request and reply) this might be too much overhead. Because of this DNS is usually done with UDP but has built retries on top of UDP.
share|improve this answer
2  
You should also have a look at your 80% packet loss with UDP. It looks like your hardware is not fast enough to process the packets in the same speed they get send. While TCP adapts to this kind of packet loss with slowing down, UDP will just send at the same speed and continue to loose packets. But at the end it is not relevant how fast you can send, but what you receive. –  Steffen Ullrich Mar 30 at 19:56
1  
Something else that might be a factor is TCP acceleration/offloading to the network card (if it supports it). –  cpugeniusmv Mar 30 at 20:03
1  
Packet sending might be more efficient than receiving, especially if the last one is interrupt driven. –  Steffen Ullrich Mar 30 at 20:11
1  
people also use UDP for an embedded device to broadcast the data it is collecting over a wire and not bother with the connection setup –  ratchet freak Mar 30 at 22:31
3  
You are most likely IO bound by the PCI express bus. The network cards will have TCP segment offloading enabled, most likely. This means that TCP transfers will be sent to the card asone big block, then the card slices and dices them into packets and puts them on the wire. There is no equivalent for UDP, so the result is one PCIe transaction (and all associated overheads) for each packet. –  alex.forencich Mar 31 at 2:54

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.