1

I'm running a system off of a battery, so I care a lot about cumulative CPU time usage.

A process that runs for a long time accumulates a lot of CPU time in top compared to a process that runs and quits a lot, so top's TIME+ column isn't very helpful to me.

I'd like to record the CPU time of all processes of a command over a long time, then add it all up and display it in a row. I'd do this for every command on the system that actually gets run. For example:

bash            5:00:00
dbus-daemon     4:30:00
firefox          3:00:00
NetworkManager  1:20:00

It is arranged into one command per row—not one process per row. The CPU time of all processes of a command is added together to produce the right column.

bash would never be as high as I've placed it if only the CPU time of a single process were counted, but after all the processes are added together, it might be that high. Bash is run a lot, every time there's a shell script with a #!/bin/bash shebang.

2 Answers 2

1

A good view on this is can be easily obtained with the atop command with the -p option, which shows the cumulative CPU (or rather resources) per command, i.e. for all ssh invocations.

It's showing the information in certain intervals similar to the normal top, but with a higher interval (default is 10s) this should be a fine fit for the problem. Running atop -p 60 for example gives the summary for a full minute, which should be fine in most cases.

1
  • This looks amazing! I hope to exorcise some power-hungry daemons on a Pinephone Pro using this. Commented Jan 12 at 19:31
0

My gut feeling is that you'll learn more from powertop than from actually counting process times, but anyways:

Install bpftrace; it comes with the execsnoop.bt example (which probably gets installed into /usr/share/bpftrace/tools). Add another probe for tracepoint:syscalls:sys_enter_exit*, save the starting time of each PID and print the difference at the exit; something like (this is very untested, not at my favourite machine for that right now, and only strives to illustrate):

#!/usr/bin/bpftrace
tracepoint:syscalls:sys_enter_exec*
{
    @start[pid] = nsecs;
    printf("START;%-6d;", pid);
    join(args->argv);
}
tracepoint:syscalls:sys_enter_exit*
{
    $from = @start[pid];
    $until = nsecs;
    printf("STOP;%-5d;%-16d\n", pid, $until-$from);
}

make that executable, run as root, and save output to file³.

Analyzing the output will probably be a bit annoying, as bpftrace (to the best of my knowledge) has no great functionality to avoid things like line breaks in printed program names and arguments. But essentially, you start by parsing a line for START; or STOP;, followed by a PID. The line ends where the next line actually starts with START; or STOP;, because join(argv) can and will contain line breaks.

Armed with that, you can accumulate run times per executable.

I'm running a system off of a battery, so I care a lot about cumulative CPU time usage.

That helps probably less than you think – CPU time at 800 MHz eats less battery than the same CPU time at 1600 MHz; whether a given problem needs half, more or less than half the time that is does at 800 MHz depends on the problem.

Worse even: a program optimized for spending little CPU time will do things like yielding very often, but only after relying on the operating system waking it⁰ soon. In a low-load situation, that's great, it allows CPU cores to go to sleep in between, in a high-load situation it's terrible, because suddenly you both get to deal with a flood of unnecessary context switches, and the things that make your CPU spend less energy at the same clock rates for the same task¹ get thrown off the racks.

And: especially in embedded use cases, the amount of time spend handling interrupts and doing other kernel work will be relatively relevant to power usage, and that's something you're not observing at all by this.

I, however, like your example a lot!

bash would never be as high as I've placed it if only the CPU time of a single process were counted, but after all the processes are added together, it might be that high. Bash is run a lot, every time there's a shell script with a #!/bin/bash shebang.

Hm, but bash would also spend almost no CPU time. It would do a tiny bit of parsing, then invoke some other process and wait until it's done. It would, down to a rounding error, either wait for input or wait for commands to finish, unless you have very inefficiently written shell scripts². So, all the time that a shell script actually takes to execute would be spend in other programs than bash.


⁰ setting a timer, poll/selecting a frequently changed fd, initiating a tiny network transfer…

¹ That being mostly lower caches being warm , i.e. practically no memory fetches occurring, and branch prediction being right most of the time, so that speculative execution can be reduced to the most likely case

² for example, trying to do a large mathematical problem with a long loop in bash

³ e.g., chmod script.bt; sudo ./script.bt > lifetimes.log

4
  • Are you measuring CPU time or real time? Commented Sep 20, 2023 at 13:46
  • ah, good catch, that script would be measuring real time Commented Sep 20, 2023 at 13:47
  • Powertop would measure power better, but can powertop collect cumulative per-command stats like what I'm after? Commented Sep 20, 2023 at 13:59
  • sure, powertop -C -t 10 writes a CSV report about 10s of observation, and you can then consolidate with python, mlr or any language of your liking Commented Sep 20, 2023 at 14:07

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .