1

When reading this Kubernetes blog post about Intel's CPU manager it mentions that you can avoid cross-socket traffic by having CPUs allocated on the socket near to the bus which connects to an external device.

What does cross-socket traffic mean and what problems can it cause? These are my guesses:

  • A CPU from one socket needs access to a device connected to a bus that is only accessible to CPUs in another socket, so instructions to that device must be written to memory to be executed by a CPU in this other socket
  • A CPU from one socket needs access to a device connected to a bus that is only accessible to CPUs in another socket, so instructions to that device are sent directly to a CPU in this other socket to be forwarded to the device (not sure if this is even possible)
5

The authors of the Kubernetes blog post just speak gibberish trying to reinvent the wheel - one more PBS (portable batch system), which they call "CPU manager".

Answering question: "What does cross-socket traffic mean and what problems can it cause?" - it's necessary to say first, that this is about Multiprocessor Computers, i.e. computer systems with two or more CPUs and CPU sockets respectively. Multiprocessor systems are available in two different architectures: SMP (symmetric multiprocessing) and AMP (asymmetric multiprocessing).

Most of multiprocessor systems available at the moment are SMP architecture systems. Such the systems have so called shared memory which is visible to indepentent physical CPUs as common main memory. There are two types of such the systems according to type of physical CPUs interconnection: system bus and crossbar switch.

Diagram of SMP system with crossbar switch:

SMP crossbar switch

Diagram of SMP system with system bus:

SMP system bus

Mostly SMP systems have system bus type CPUs connection and the Kubernets blog post is about systems of such the kind.

SMP systems with system bus CPUs connection have both advantages and disadvantages. The most significant disadvantage of this systems is that they're NUMA (non-uniform memory access) systems. What does it mean. Every CPU socket physically associates its own memory bank, but Linux kernel can't distiungish this assosiation in SMP - the memory banks are seen to Linux as single integral memory. But despite this fact, NUMA phenomenon arises - interoperation of a physical CPU with addresses of its own physical memory bank isn't same fast as with memory bank(-s) assosiated with another CPU socket(-s). Thus, we wish naturally to avoid using by a physical CPU such addresses in the common main memory of SMP which belong to physical memory bank connected to another physical CPU.

The part "Limitations" of the Kubernates blog post refers to NUMA phenomenon as to "cross-socket traffic" problem (citation):

Users might want to get CPUs allocated on the socket near to the bus which connects to an external device, such as an accelerator or high-performance network card, in order to avoid cross-socket traffic. This type of alignment is not yet supported by CPU manager.

By the way, the disability to assign a thread to some definite CPU which "is closer" to something is quite natural. Linux kernel sees all of CPU cores of physical CPUs as equal ordinary SMP processors since it can't distinguish physical CPUs of SMP computer. There're some poor attempts to avoid using CPU cores which are "farther" using "warm cache" and "cold cache" signs, but it doesn't work effectively due to nature of SMP systems.

Please, read additionally:

  • @dippynark You're welcome! – Bob Jul 27 '18 at 18:29
  • I would like to point out that the diagram you give of a system-bus style architecture is not a NUMA arrangement, it's UMA. NUMA would be the memory hanging off of either the CPU's themselves, or off of their caches, such that you have to go through more than just the system-bus to get to the memory. Other than that though, great answer. – Austin Hemmelgarn Jul 27 '18 at 19:22
  • @AustinHemmelgarn Thank you so much for your feedback! Please, take into consideration that the diagram is logical diagram of SMP bus type system, not a physical. The goal of the logical diagram is to represent elements logical interconnect and the fact the memory is shared, i.e. to reflect that several physical memory banks is seen as single memory to OS. This is about "how Linux sees the SMP system" and not about "how physical components interact each with other". – Bob Jul 28 '18 at 3:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.