How to get CPU Percentage as a Counter?

Question

I would like to monitor total CPU utilization percentage as a counter. The reason I would like it as a counter is that data won't be lost between samples (and I can have the graphing side calculate the rate).

My initial approach was to use /proc/uptime with the formula (uptime-(idle_time/num_core))*100. This generally seems to be accurate across a large number of servers (something like 98% of the time), but sometimes I seem to get erroneous results. For example the following seems to suggest that there was negative CPU usage, which doesn't really make sense:

[root@ny-lb05 ~]# echo -e "scale=10\n ($(cut -f1 -d' ' /proc/uptime)-($(cut -f2 -d' ' /proc/uptime)/16))*100" | bc
5646895.3750000000
[root@ny-lb05 ~]# echo -e "scale=10\n ($(cut -f1 -d' ' /proc/uptime)-($(cut -f2 -d' ' /proc/uptime)/16))*100" | bc
5646891.5625000000

On this server I'm running:

Linux ny-lb05.ds.stackexchange.com 2.6.32-431.11.2.el6.x86_64 #1 SMP Tue Mar 25 19:59:55 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

Does someone see an error in this method of calculation? Is there a better way to get CPU utilization as a counter?

Update:
So what I'm after is the total utilization time as a monotonically increasing counter. I would expect that total utilization should never decrease. But that seems to be the case with the following: [root@ny-lb05 ~]# read uptime idle </proc/uptime; echo -e "scale=1000\n ($uptime*16-($idle))" | bc 903874.23 [root@ny-lb05 ~]# read uptime idle </proc/uptime; echo -e "scale=1000\n ($uptime*16-($idle))" | bc 903870.29

Also, according to /proc/cpuinfo, cores=siblings so I believe HT is not enabled.

Update 2:
TLDR; /proc/uptime is bugged, use /proc/stat instead.

Answer 1

(uptime-(idle_time/num_core))

May give an idea of how long the system has been busy, in seconds. Multipling that by 100 makes it centiseconds -- is that your intention?

IMO it would make more sense to consider how many processor seconds in total were available, and subtract the idle time from that:

uptime * num_core - idle_time = total active processor seconds

A utilization metric might be:

active seconds / (uptime * num_core)

E.g., if the system has been up for 10 seconds on 4 cores with 5 seconds of idle_time:

(10 * 4 - 5) / (10 * 4) = 0.875

87.5% utilization.

Or:

(10 - 5 / 4) / 10 = 0.875

Same thing, saves an operation.

---

Is there a better way to get CPU utilization as a counter?

I've done this in a system diagnostics C++ library by parsing the first line of /proc/stat, which is a combined total for all cores. The first three fields are user time, low priority (aka nice) time, and system time. The total of these is the amount of active time (note the unit here is not seconds, see /proc/stat under man proc).

If you poll this over 5 seconds, assuming a USER_HZ of 100, where total_a is the first sample (user + nice + sys) and total_b is the second sample:

(total_b - total_a) / 5 / 100 / num_cores = usage ratio

If you multiply that by 100, you have a percentage indicating an average over the 5 second interval.

Here's the logic:

• total_b - total_a = active time between samples

• Divided by the duration of the sample, 5 seconds.

• Divided by the units per second of the measurement (USER_HZ)

• Divided by the number of cores

USER_HZ is almost certainly 100. To check:

#include <stdio.h>
#include <unistd.h>

int main (void) {
    printf (
        "%ld\n", 
        sysconf(_SC_CLK_TCK)
    );

    return 0;
}

Compile: gcc whatever.c, run ./a.out.

It will be hard to get an accurate duration for this with shell tools, so you could either keep an increasing measure of the total active time (I think that is your intention) or use a fairly long interval, e.g. 30+ seconds.

Answer 2

You problem is probably because you are reading /proc/cputime from two separate processes. The idle time will will increase slightly between each cat, giving the possibility for a lower reading the second time. I recommend doing this instead:

read uptime idle </proc/cputime
echo -e "scale=10\n ($uptime-($idle/16))*100" | bc

Also, if you want your results to be a total utilisation percentage, then you should divide by the uptime again:

read uptime idle </proc/cputime
echo -e "scale=10\n ($uptime-($idle/16))/$uptime*100" | bc