I am doing some performance troubleshooting of some long running jobs which start processes that consume entire CPU cores, but are single-threaded. I observed these processes bouncing around to different CPUs continually. Why would the same process move around to different CPUs? It doesn't seem normal that the kernel scheduler would move running processes around that frequently, or at all. Yet, I see this behavior on many different processes. It seems like the more active the process is, the more it changes around to different CPUs.
I observe this behavior with top
. I add the Last Used Cpu
column. I then watch the interesting processes change to different CPUs continually even though the process ID remains the same.
It moves around so much that I believe I am seeing inconsistent performance on certain jobs, because the processes often collide with each other on the same CPU temporarily, even though they should all be balanced across different CPUs while the overlying jobs are running. The server is mostly idle except for these processes while these jobs run. So, I would expect Job 1 to start processes on one set of CPUs, and Job 2 to end up on a different set of CPUs and stay there.
Amazon Linux 2 Kernel 4.14.x on EC2