2

I have a developer complaining that the CPU usage reported by /proc/stat is not consistent.

It is my understanding that /proc/stat is counting ticks, which since kernel 2.6 has been fixed to 100/s

Now I have made an observation on CPU1 that /proc/stat it may count more than 100 ticks during a second, which shouldn't be possible.

This script shows how I do the calculation

cat /proc/uptime
b=`awk /cpu1/'{print $2+$3+$4+$5+$6+$7+$8+$9+$10+$11}' /proc/stat`
sleep 1
a=`awk /cpu1/'{print $2+$3+$4+$5+$6+$7+$8+$9+$10+$11}' /proc/stat`
cat /proc/uptime
expr $a - $b

Example output

32.80 19.06
33.86 19.51
CPU1 jiffies/s 137

I sum up the fields in /proc/stat, wait a second and do it again. Then I sometimes get a sum difference of approximately 140. It is quite stable, may go a few counts up and down.

CONFIG_HZ is 100 for the kernel. If I compile with 250 or 1000, the variation becomes smaller, but it is still there.

It seems to be the user space and the idle fields that counts most of the jiffies. In one sample I calculated 102 user space jiffies, which can be explained by sleep taking 20 ms extra. Thus user space takes up all the time. But then there was also 33 idle ticks, which were executed on a already fully booked CPU.

I have a dual core ARMv7 processor running Linux kernel 4.14.34 with RT preempt patches. Together with that I'm running a real time control application.

My problem is that /proc/stat figures are inconsistent. But to understand that, I very much would like to understand why /proc/stat can count above 100.

Edit: Added cat /proc/uptime in script

0

The script you use is the main issue. Context switching is very expensive in term of latency. sleep 1 puts your script in sleeping state. when the timer is finished the processus is becoming runable again, may be also being reallocated to another core of your cpu (migration). So basically you are not polling /proc/stat exactly every 1000ms as far you can state it in your post.

This is by the way not a very good way to assess a RT kernel.

I really invite you to use for exemple sysjitter from Solarflare sysjitter openonload which will gives you far better metrics on how the kernel is ticking.

The main concept, is to have an infinite loop pinned to one core (1 running thread per core) which pool Time Stamp counter(TSC) register of the core, and comparing two following value to see the difference wwhich can occurs each time kernel is preempting the running process (supposedly 100hz).

This way you will have a better vision of the system jitter of your server, and decide may be to tune it a little bit more.

Also you should have a look to cpu isolation, to have dedicated core for your application, and dynamic ticking kernel configuration, if your use case requires such improvment.

Edit: I suggest you to only use awk for all your needs with little script like this one :

#!/usr/bin/awk 

BEGIN { 
    sum1=0;
    sum2=0;
    result=0;
    file="/proc/stat";



while (( getline < file ) > 0 ) {
    if ($1=="cpu1"){ sum1=$2+$3+$4+$5+$6+$7+$8+$9+$10+$11 }; 
    } 
    close(file) ;
    print  "firt time " sum1;
    system("sleep 10");

while (( getline < file ) > 0 ) {
        if ($1=="cpu1") { sum2=$2+$3+$4+$5+$6+$7+$8+$9+$10+$11 }
    }
    close (file)
    print "secont time " sum2;
    result=sum2-sum1;
    print "sum is " result/10;
}

this samples over 10 seconds and divides the result by ten to get average time for 1s slices.

| improve this answer | |
  • Agree that latency on sleep can affect. I added a cat /proc/uptime in my script. This shows that latency cannot be more than 60 ms. This would however only account for 6 ticks out of 40 ticks too many. I will try to look at sysjitter, it sounds like a good principle.Thanks for that. – Kjeld Flarup Aug 8 '19 at 9:40
  • you can also check your power management. which can contribute a lot to induce latency when changing power level of core. – netmonk Aug 8 '19 at 9:50
  • Power management is also a good suggestion. However the hardware does not implement that. I also see that the ticks vary independently on the two CPU kernels. – Kjeld Flarup Aug 9 '19 at 10:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.