11

We're running on a virtual "dedicated" server, which should, in theory, mean that we're the only guys on the server. In practice.... I'm thinking we might not be.

enter image description here

Notice that although it looks like we're killing our machine, "Steal time" is at 71%

I'm taking statistics on load and I was disappointed that this stat didn't show up in my graphs. Are there any tools which monitor this which might be able to help?


Additional information:

We're running 4 cores, model:

# grep "model name" /proc/cpuinfo | sort -u
model name  : Intel(R) Core(TM)2 Duo CPU     E7500  @ 2.93GHz
9
  • 1
    Virtual dedicated? In case of XEN they would need to pin dedicated cores for dedicated use in your VM. Looks like your provider has overbooked CPUs by a unfair amout. What does he say to this?
    – Nils
    Jun 20, 2012 at 18:56
  • 1
    How many vCPUs have you got and what type of CPU is reported in grep "model name" /proc/cpuinfo|sort -u? If this really a dedicated server then there is something eating up CPU-time on the Dom0. OR they gave you more vCPUs than are available in the Dom0.
    – Nils
    Jun 20, 2012 at 19:33
  • 1
    Unless this is a momentary outlier, it looks like your isp is lieing to you and they are, in fact, running other cpu heavy vms on this machine, or there is something configured very wrong that is causing dom0 to hog a lot of cpu time.
    – psusi
    Jun 21, 2012 at 3:33
  • 1
    SuSE recommends to reserve two cores solely for the Dom0 so it can do all IO-handling without bothering other VMs. In my eyes that would only be necessary for systems with stolen time in the DomUs AND heavy IO traffic. I wanted to know if your provider assigned more vCPUs than logical cores - like assign 4 vCPUs while only 2 logical CPUs are available in the Dom0 - that would explain "stolen", too (and is a pretty braindead idea - but possible in XEN).
    – Nils
    Jun 21, 2012 at 20:55
  • 1
    The root cause of this one turned out to be that the ISP had the VM incorrectly configured. The guest was being told it had more cores than it actually did. This seemed to cause havoc with the scheduling. The ISP couldn't provide intelligent tech support, but we were able to "prove" the problem by disabling odd-numbered cores in /proc. Never a problem since.
    – mgjk
    Feb 23, 2014 at 13:15

3 Answers 3

13

You're question is well defined, but you're not giving a lot of information about your environment, how you're currently monitoring or what graphing tools you're using. However, given that SNMP is used pretty much universally for that I'll assume that you're using it and have at least some familiarity with it.

Although (as near as I can tell) the CPU Steal time isn't currently available from snmpd, you can extend it yourself with the UCD-SNMP-MIB::extOutput object and exec commands.

The easiest way (that I've found) to get the steal time is from iostat. Using the following construct we can get just the steal time:

$ iostat -c | awk 'NR==4 {print $5}'
0.00

Therefore, append the following to your snmpd.conf:

exec cpu_steal_time /usr/bin/iostat -c | /usr/bin/awk 'NR==4 {print $5}'

(Alternatively you can put the command in a wrapper script and call the wrapper from inside snmpd.conf.)

Each exec call in snmpd.conf is indexed starting from 1. So if you only have a single exec statement then you'll want to poll UCD-SNMP-MIB::extOutput.1. If this is the 5th exec statement then poll UCD-SNMP-MIB::extOutput.5, etc.

The numeric OID for UCD-SNMP-MIB::extOutput is .1.3.6.1.4.1.2021.8.1.101 so if you're at index 1 it would be .1.3.6.1.4.1.2021.8.1.101.1, and index 5 would be .1.3.6.1.4.1.2021.8.1.101.5, etc.

You then create a graph polling that SNMPD OID of type gauge, ranging from 0–100. This should give you some pretty graphs.

3
  • Great answer. How often will these statics be gathered? Just during poll, or is there a way like in the RMON-MIB that will record values without external poll?
    – Nils
    Jun 20, 2012 at 19:36
  • I believe it would pull this each time the snmpd is queried for that OID.
    – bahamat
    Jun 20, 2012 at 21:47
  • If iostat is not installed: top -bn1 | sed -nr '3s/.*,//gp'
    – davide
    Nov 25, 2014 at 1:51
10

sar -u might be helpful in your case. sar is normally part of the sysstat-package.

1
  • I wish I could set more than one answer as the accepted answer. Both answers have been very useful :-) Thank you!
    – mgjk
    Jun 22, 2012 at 15:27
1

The most upvoted answer is great, but at this time it isn't fully-working: net-snmp loses pipe in exec call, so this should be looking like this

extend-sh cpu_steal_time /usr/bin/iostat -c 1 1 | /usr/bin/awk '!/%user|Linux|^$/ {print $5}'

And the result will be visible under nsExtendOutput1Table:

# snmpwalk localhost NET-SNMP-EXTEND-MIB::nsExtendOutput1Table
NET-SNMP-EXTEND-MIB::nsExtendOutput1Line."cpu_steal_time" = STRING: 0.60
NET-SNMP-EXTEND-MIB::nsExtendOutputFull."cpu_steal_time" = STRING: 0.60
NET-SNMP-EXTEND-MIB::nsExtendOutNumLines."cpu_steal_time" = INTEGER: 1
NET-SNMP-EXTEND-MIB::nsExtendResult."cpu_steal_time" = INTEGER: 0

where nsExtendOutput1Line oid is .1.3.6.1.4.1.8072.1.3.2.3.1.1:

snmpwalk localhost .1.3.6.1.4.1.8072.1.3.2.3.1.1
NET-SNMP-EXTEND-MIB::nsExtendOutput1Line."cpu_steal_time" = STRING: 0.60

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .