0

This is a bit of a puzzler. I have 28 VMs, running on ESXi 5.5, each with 6 vcpus assigned to them. They're not particularly loaded, and they're not running anything other than a single java process each. When I run top on any of them, and display individual CPU output, they all essentially look like this:

top - 21:00:15 up 6 days, 12:52,  1 user,  load average: 0.89, 0.85, 0.82
Tasks: 110 total,   1 running, 108 sleeping,   0 stopped,   1 zombie
Cpu0  :  3.7%us,  0.3%sy,  0.0%ni, 96.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu1  : 38.4%us,  3.0%sy,  0.0%ni, 51.9%id,  0.0%wa,  1.3%hi,  5.4%si,  0.0%st
Cpu2  :  2.7%us,  0.3%sy,  0.0%ni, 96.6%id,  0.0%wa,  0.0%hi,  0.3%si,  0.0%st
Cpu3  :  3.7%us,  0.7%sy,  0.0%ni, 95.6%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu4  :  3.4%us,  0.3%sy,  0.0%ni, 96.0%id,  0.0%wa,  0.0%hi,  0.3%si,  0.0%st
Cpu5  :  4.0%us,  0.7%sy,  0.0%ni, 95.0%id,  0.0%wa,  0.0%hi,  0.3%si,  0.0%st
Mem:  12168008k total,  7323476k used,  4844532k free,   405332k buffers
Swap:  4194296k total,        0k used,  4194296k free,  1198540k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
20543 root      18   0 5704m 303m  18m S 64.6  2.6 555:03.29 java
25873 root      15   0 90152 3336 2604 S  0.3  0.0   0:00.02 sshd
25952 root      15   0 12760 1276  944 R  0.3  0.0   0:00.01 top


On every one of the servers, Cpu1 is the only cpu shown with any load. It never moves to any other cpu. This was noticed two weeks ago. All have been rebooted since, and the behavior is still the same. There's no affinity set for the process:

# taskset -p $(pgrep java)
pid 20543's current affinity mask: 3f

If I watch the individual threads in a non-sleep state with something like:

# cd /proc/$(pgrep java)/task
# while true; do awk '($3 != "S") { print "LWPID: " $1 " CPU: " $39}' */stat; sleep 5; echo -; done

They seem to hop around to assorted CPUs.

# cat /etc/redhat-release
CentOS release 5.7 (Final)
# uname -a
Linux ----- 2.6.18-274.18.1.el5 #1 SMP Thu Feb 9 12:45:44 EST 2012 x86_64 x86_64 x86_64 GNU/Linux
# java -version
java version "1.7.0_13"
Java(TM) SE Runtime Environment (build 1.7.0_13-b20)
Java HotSpot(TM) 64-Bit Server VM (build 23.7-b01, mixed mode)
# top --version
        top: procps version 3.2.7

Anyone have a clue why they might always show up on a single CPU?

  • What is the actual problem here? If the load is significantly below 1 (as it appears to be) it won't help anything to spread it. So please tell us why you feel this is worth investigating. – DepressedDaniel Jan 30 '17 at 3:20
  • Simple curiosity. I eventually rationalized it in that they are all identical, and the behavior was deterministic because they all start the same way. I don't have visibility into the code but I assume it was probably the main event loop that was occupying the cpu.They've since been moved to RH 6, and are much heavier loaded these days. One or more factors in that switch have resulted in different behavior. – Allen Garvin Jan 30 '17 at 17:58
0

It could be a preference by the machine to only have one CPU awake when it doesn't actually need to spread the load.

The each process would see an awake CPU with time to spare and prefer it.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.