In /proc/<pid>/stat I add user cpu, system cpu, child user and child system cpu together for all processes. I take a delta from a previous sample.

Immediately after I sum the user, nice and system CPU from /proc/stat which should be for the entire box. Again I take a delta.

The sum from the processes is almost always slightly greater than that for the overall processor and I can't figure out why.

  • I ran another test where I drove a single process to have 100% CPU on one of the four cores and measured just that process and /proc/stat. The box was otherwise idle. The process registered 100 clock ticks per second normally (expected) and occasionally 101 ticks. When it registered 101, that extra one never accumulated in /proc/stat which kept going up 100 ticks every second. It appears it sometimes misses them.
    – user331809
    Jan 17 '19 at 15:23

There are a few reasons why that would be the case, the most obvious one is that the per-process accounting uses different logic for counting different things.

Two specific examples:

  1. The system stat has a nice column that counts user time spent in nice tasks (lower priority) - which you didn't specify that you use, so I'm guessing you didn't sum it. In the process this will be counted as standard user time.
  2. iowait time - which also you didn't account for - is likely counted in the process stat as system time. But its a bad idea to also count in iowait time as it is very unreliable account of actual time, as man proc explains:

iowait (since Linux 2.5.41)

(5) Time waiting for I/O to complete. This value is not reliable, for the following reasons:

  1. The CPU will not wait for I/O to complete; iowait is the time that a task is waiting for I/O to complete. When a CPU goes into idle state for outstanding task I/O, another task will be scheduled on this CPU.

  2. On a multi-core CPU, the task waiting for I/O to complete is not running on any CPU, so the iowait of each CPU is difficult to calculate.

  3. The value in this field may decrease in certain conditions.

Lastly, I have to note that I believe this type of accounting is useless as it can never be accurate enough for any serious purpose - system and process counters are processed in different parts of the system for different purposes and will never match to any useful degree.

  • Thank you, I suspect the last paragraph is the answer sadly though it boggles me that we just skip clock ticks sometimes. As for nice, yes I did say that I added it. And iowait, not added but I actually checked all the counters to make sure nothing unexpected was getting incremented and it was not. It would be a bug if this was counted as real CPU in the process.
    – user331809
    Jan 18 '19 at 18:08
  • As far as I understand, iowait (D state in ps) is counted as process system time.
    – Guss
    Jan 18 '19 at 19:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.