2

I would like to monitor total CPU utilization percentage as a counter. The reason I would like it as a counter is that data won't be lost between samples (and I can have the graphing side calculate the rate).

My initial approach was to use /proc/uptime with the formula (uptime-(idle_time/num_core))*100. This generally seems to be accurate across a large number of servers (something like 98% of the time), but sometimes I seem to get erroneous results. For example the following seems to suggest that there was negative CPU usage, which doesn't really make sense:

[root@ny-lb05 ~]# echo -e "scale=10\n ($(cut -f1 -d' ' /proc/uptime)-($(cut -f2 -d' ' /proc/uptime)/16))*100" | bc
5646895.3750000000
[root@ny-lb05 ~]# echo -e "scale=10\n ($(cut -f1 -d' ' /proc/uptime)-($(cut -f2 -d' ' /proc/uptime)/16))*100" | bc
5646891.5625000000

On this server I'm running:

Linux ny-lb05.ds.stackexchange.com 2.6.32-431.11.2.el6.x86_64 #1 SMP Tue Mar 25 19:59:55 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

Does someone see an error in this method of calculation? Is there a better way to get CPU utilization as a counter?

Update:
So what I'm after is the total utilization time as a monotonically increasing counter. I would expect that total utilization should never decrease. But that seems to be the case with the following: [root@ny-lb05 ~]# read uptime idle </proc/uptime; echo -e "scale=1000\n ($uptime*16-($idle))" | bc 903874.23 [root@ny-lb05 ~]# read uptime idle </proc/uptime; echo -e "scale=1000\n ($uptime*16-($idle))" | bc 903870.29

Also, according to /proc/cpuinfo, cores=siblings so I believe HT is not enabled.

Update 2:
TLDR; /proc/uptime is bugged, use /proc/stat instead.

  • @TAFKA'goldilocks': According to your answer uptime * num_core - idle_time = total active processor seconds. Doesn't it logically follow that total active processor seconds should never decrease? – Kyle Brandt Apr 9 '14 at 13:53
  • Yeah, you're right, sorry. There can't have been more idle time than total time since the last sample (comment deleted). – goldilocks Apr 9 '14 at 13:56
  • BTW, it terms of calculating a current usage % ala top, I've done this in the past using fields from /proc/stat; the first cpu is a combined total, which is useful (then you have breakdowns for each individual core). You then need two samples to determine usage in relation to a unit of time. – goldilocks Apr 9 '14 at 13:58
  • @TAFKA'goldilocks': Good idea, I'll do the math from /proc/stat and see if I run into this same bug on the machine exhibiting the behavior. – Kyle Brandt Apr 9 '14 at 14:02
  • @TAFKA'goldilocks': Do you what subset of the /proc/stat columns summed equals utilization? Is it user/system/nice? – Kyle Brandt Apr 9 '14 at 14:21
4

(uptime-(idle_time/num_core))

May give an idea of how long the system has been busy, in seconds. Multipling that by 100 makes it centiseconds -- is that your intention?

IMO it would make more sense to consider how many processor seconds in total were available, and subtract the idle time from that:

uptime * num_core - idle_time = total active processor seconds

A utilization metric might be:

active seconds / (uptime * num_core)

E.g., if the system has been up for 10 seconds on 4 cores with 5 seconds of idle_time:

(10 * 4 - 5) / (10 * 4) = 0.875

87.5% utilization.

Or:

(10 - 5 / 4) / 10 = 0.875

Same thing, saves an operation.


Is there a better way to get CPU utilization as a counter?

I've done this in a system diagnostics C++ library by parsing the first line of /proc/stat, which is a combined total for all cores. The first three fields are user time, low priority (aka nice) time, and system time. The total of these is the amount of active time (note the unit here is not seconds, see /proc/stat under man proc).

If you poll this over 5 seconds, assuming a USER_HZ of 100, where total_a is the first sample (user + nice + sys) and total_b is the second sample:

(total_b - total_a) / 5 / 100 / num_cores = usage ratio

If you multiply that by 100, you have a percentage indicating an average over the 5 second interval.

Here's the logic:

  • total_b - total_a = active time between samples

  • Divided by the duration of the sample, 5 seconds.

  • Divided by the units per second of the measurement (USER_HZ)

  • Divided by the number of cores

USER_HZ is almost certainly 100. To check:

#include <stdio.h>
#include <unistd.h>

int main (void) {
    printf (
        "%ld\n", 
        sysconf(_SC_CLK_TCK)
    );

    return 0;
}

Compile: gcc whatever.c, run ./a.out.

It will be hard to get an accurate duration for this with shell tools, so you could either keep an increasing measure of the total active time (I think that is your intention) or use a fairly long interval, e.g. 30+ seconds.

  • This seems to work. I'm using opentsdb, which does a rate calculation for me from counters, so all I need to do is send it user+nice+system from the first line of /proc/stat divided by the number of cores. Whatever bug is messing with /proc/uptime doesn't seem to impact /proc/stat. – Kyle Brandt Apr 9 '14 at 15:13
0

You problem is probably because you are reading /proc/cputime from two separate processes. The idle time will will increase slightly between each cat, giving the possibility for a lower reading the second time. I recommend doing this instead:

read uptime idle </proc/cputime
echo -e "scale=10\n ($uptime-($idle/16))*100" | bc

Also, if you want your results to be a total utilisation percentage, then you should divide by the uptime again:

read uptime idle </proc/cputime
echo -e "scale=10\n ($uptime-($idle/16))/$uptime*100" | bc
  • Nope, it my actual code it is a single read: [kbrandt@ny-lb06: ~] read uptime idle </proc/uptime; echo -e "scale=10\n ($uptime-($idle/16))*100" | bc <990@8:03> 6259851.5000000000 [kbrandt@ny-lb06: ~] read uptime idle </proc/uptime; echo -e "scale=10\n ($uptime-($idle/16))*100" | bc <991@8:03> 6259842.8125000000 – Kyle Brandt Apr 9 '14 at 13:04
  • Also, the difference here would be pretty insignificant except in extremely unusual cases. – goldilocks Apr 9 '14 at 13:05
  • @TAFKA'goldilocks', the difference is only 50ms, which is about right for the overhead of cut, especially is most of the processors on the system are idle. – Graeme Apr 9 '14 at 13:09
  • That's why I'm saying it's insignificant, unless the system has only been up for a few hundred milliseconds. It is more efficient to do it in one sample, but the OP's problem here is mostly his metric (I notice we've both come to the same conclusion on that). – goldilocks Apr 9 '14 at 13:12
  • The metric I'm going for is a monotonically increasing counter of utilization, so on the query side I can take the deltas, divide by total seconds, and get a rate. – Kyle Brandt Apr 9 '14 at 13:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.