My gut feeling is that you'll learn more from powertop than from actually counting process times, but anyways:
Install bpftrace; it comes with the execsnoop.bt example (which probably gets installed into /usr/share/bpftrace/tools). Add another probe for tracepoint:syscalls:sys_enter_exit*, save the starting time of each PID and print the difference at the exit; something like (this is very untested, not at my favourite machine for that right now, and only strives to illustrate):
#!/usr/bin/bpftrace
tracepoint:syscalls:sys_enter_exec*
{
@start[pid] = nsecs;
printf("START;%-6d;", pid);
join(args->argv);
}
tracepoint:syscalls:sys_enter_exit*
{
$from = @start[pid];
$until = nsecs;
printf("STOP;%-5d;%-16d\n", pid, $until-$from);
}
make that executable, run as root, and save output to file³.
Analyzing the output will probably be a bit annoying, as bpftrace (to the best of my knowledge) has no great functionality to avoid things like line breaks in printed program names and arguments. But essentially, you start by parsing a line for START; or STOP;, followed by a PID. The line ends where the next line actually starts with START; or STOP;, because join(argv) can and will contain line breaks.
Armed with that, you can accumulate run times per executable.
I'm running a system off of a battery, so I care a lot about cumulative CPU time usage.
That helps probably less than you think – CPU time at 800 MHz eats less battery than the same CPU time at 1600 MHz; whether a given problem needs half, more or less than half the time that is does at 800 MHz depends on the problem.
Worse even: a program optimized for spending little CPU time will do things like yielding very often, but only after relying on the operating system waking it⁰ soon. In a low-load situation, that's great, it allows CPU cores to go to sleep in between, in a high-load situation it's terrible, because suddenly you both get to deal with a flood of unnecessary context switches, and the things that make your CPU spend less energy at the same clock rates for the same task¹ get thrown off the racks.
And: especially in embedded use cases, the amount of time spend handling interrupts and doing other kernel work will be relatively relevant to power usage, and that's something you're not observing at all by this.
I, however, like your example a lot!
bash would never be as high as I've placed it if only the CPU time of a single process were counted, but after all the processes are added together, it might be that high. Bash is run a lot, every time there's a shell script with a #!/bin/bash shebang.
Hm, but bash would also spend almost no CPU time. It would do a tiny bit of parsing, then invoke some other process and wait until it's done. It would, down to a rounding error, either wait for input or wait for commands to finish, unless you have very inefficiently written shell scripts². So, all the time that a shell script actually takes to execute would be spend in other programs than bash.
⁰ setting a timer, poll/selecting a frequently changed fd, initiating a tiny network transfer…
¹ That being mostly lower caches being warm , i.e. practically no memory fetches occurring, and branch prediction being right most of the time, so that speculative execution can be reduced to the most likely case
² for example, trying to do a large mathematical problem with a long loop in bash
³ e.g., chmod script.bt; sudo ./script.bt > lifetimes.log
