2

In one of my KVM setups I am seeing high CPU stealing for Core 0 of both the guests. I have gone through saveral web results related to cpu steal but no were core steal is explained.

If I take average it doesn't look that bad. 40/8 ~ 5%. But for single core its very high.

Threads: 815 total, 4 running, 811 sleeping, 0 stopped, 0 zombie %Cpu0 : 18.4 us, 0.8 sy, 0.0 ni, 35.2 id, 0.0 wa, 0.0 hi, 6.4 si, 39.2 st %Cpu1 : 38.6 us, 6.9 sy, 0.0 ni, 48.5 id, 5.9 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu2 : 31.0 us, 20.0 sy, 0.0 ni, 47.0 id, 2.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu3 : 22.0 us, 14.0 sy, 4.0 ni, 48.0 id, 12.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu4 : 31.7 us, 18.8 sy, 0.0 ni, 48.5 id, 1.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu5 : 41.0 us, 13.0 sy, 0.0 ni, 46.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu6 : 8.1 us, 7.1 sy, 0.0 ni, 84.8 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu7 : 35.3 us, 24.5 sy, 0.0 ni, 39.2 id, 1.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu8 : 3.0 us, 5.1 sy, 0.0 ni, 91.9 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu9 : 5.9 us, 6.9 sy, 0.0 ni, 87.1 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu10 : 6.1 us, 2.0 sy, 0.0 ni, 91.9 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu11 : 10.4 us, 13.5 sy, 0.0 ni, 24.0 id, 52.1 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu12 : 50.0 us, 10.0 sy, 0.0 ni, 40.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem : 48966980 total, 13622884 free, 25813260 used, 9530836 buff/cache

I have done cpu isolation in the host and there are only two guest VM per host. cat /proc/cmdline

BOOT_IMAGE=/vmlinuz-3.10.0-693.el7.x86_64 root=/dev/mapper/VolGrp-Vol1 ro crashkernel=auto rd.lvm.lv=VolGrp/Vol1 rd.lvm.lv=VolGrp/Vol0 console=ttyS0,115200n8 isolcpus=2-15,18-31 nohz_full=2-15,18-31 rcu_nocbs=2-15,18-31 intel_idle.max_cstate=1 intel_pstate=disable nosoftlockup=0 audit=0 mce=ignore_ce transparent_hugepage=never

Also same cores are not shared between two VMs From virsh dumpxml

VM1

<vcpu placement='static' cpuset='2,4,6,8,10,12,14,18,20,22,24,26,28,30'\>13</vcpu>

VM2

<vcpu placement='static' cpuset='3,5,7,9,11,13,15,19,21,23,25,27,29,31'\>13</vcpu>

Host NUMA information:

NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30 NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31

Host OS: CentOS 7.4

Guest OS: CentOS 7.4

qemu KVM version: 1.5.3

Any suggestions who to find why i have such a high steal value ?

2

You are doing oversubscription of your virtualised environment e.g. collectively, the VMs on that host are trying to use more CPU resources than the host has.

If you pinning a couple of VMs to a single CPU, so much the better for debugging, you will be able to point out quickly for the culprit. Watch out for CPU/I/O/memory, often lack of resources on one influence the others.

see iostat - What does the 'steal' field mean?

Taken from that question:

As per man vmstat:

st: Time stolen from a virtual machine. Prior to Linux 2.6.11, unknown.

....

It’s the time the hypervisor scheduled something else to run instead of something within your VM. This might be time for another VM, or for the Hypervisor host itself. If no time were stolen, this time would be used to run your CPU workload or your idle thread.

PS Over the long term, I do advise not pinning specific VMs to specific CPUs as a global strategy. The hypervisor is capable of taking better decisions than yourself when distributing the CPU resources/load.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.