Why does the same process bounce around on different "CPUs"?

Question

I am doing some performance troubleshooting of some long running jobs which start processes that consume entire CPU cores, but are single-threaded. I observed these processes bouncing around to different CPUs continually. Why would the same process move around to different CPUs? It doesn't seem normal that the kernel scheduler would move running processes around that frequently, or at all. Yet, I see this behavior on many different processes. It seems like the more active the process is, the more it changes around to different CPUs.

I observe this behavior with top. I add the Last Used Cpu column. I then watch the interesting processes change to different CPUs continually even though the process ID remains the same.

It moves around so much that I believe I am seeing inconsistent performance on certain jobs, because the processes often collide with each other on the same CPU temporarily, even though they should all be balanced across different CPUs while the overlying jobs are running. The server is mostly idle except for these processes while these jobs run. So, I would expect Job 1 to start processes on one set of CPUs, and Job 2 to end up on a different set of CPUs and stay there.

Amazon Linux 2 Kernel 4.14.x on EC2

Answer 1

The Linux scheduler uses natural CPU affinity: the scheduler attempts to keep processes on the same CPU as long as practical for performance reasons. You can enforce CPU affinity in a normal Linux system with taskset.

But, as I understood it, unless you have dedicated instances, on EC2 there will be a lot of CPU sharing. You might be able to tie a process to a vCPU, but that's just windowdressing.

Answer 2

You wrote watching the phenomenon using top. I am not sure it is the right tool for your analysis since, as man top acknowledges :

17. P -- Last used CPU (SMP) A number representing the last used processor. In a true SMP environment this will likely change frequently since the kernel intentionally uses weak affinity. Also, the very act of running top may break this weak affinity and cause more processes to change CPUs more often (because of the extra demand for cpu time).

I would personally prefer relying on the count of Rescheduling interrupts as reported in /proc/interrupts.

Answer 3

even though they should all be balanced across different CPUs while the overlying jobs are running

No they shouldn't. You didn't say which scheduler you are using but AFAIK all current mainstream distros use the Completely Fair scheduler by default. This will assign a task to a CPU based on the available CPUs - with the CPU last used by the process being the preferred destination, a CPU in the same hyperthread group next, then a CPU on the same socket (sharing the same L2 cache IIRC?) - and then things start getting complicated when you're dealing with NUMA. Its all documented on the internet if you go looking for it.

On a happy machine, the maximum timeslot available to a process to run on a CPU will not be reached - the task will do something which means it has to wait for something to happen and hence yield its place. When the load average gets close to the number of CPUs there is contention for the CPU resource and the scheduler will start kicking tasks out of the CPU before they are ready to yeild (they are pre-empted). Typically this decreases the throughput of the system slightly.

It is possible to tweak some of the coefficients used to weight the CPU affinity algorithm. Run sysctl -A | grep "sched" | grep -v "domain" to see the parameters. But note that, to some extenet the kernel will tune CFS at run time.

The server is mostly idle except for these processes while these jobs run

Really? These processes don't do any network or storage I/O?