Noninteractively identify processes using up a lot of CPU-time

Question

I have a weird issue: sometimes when my monitor is turned off, the fans are running loud, even when there shouldn't be much usage of the CPU on the system as far as I know.

But as soon as I move my mouse and start top to try to diagnose this, the activity, whatever it is, stops; with the fans winding down.

So I want a script/program/method that I could start at some point in time, leave the computer unattended while this program is recording CPU activity of processes, then when I resume operating the computer I should be able to read the program's report from which I would quickly know what processes are making the fans work hard.

EDIT: one chromium process is the one making the fans run loud while the screen is off. No idea why, though.

Answer 1

You could just run "ps" every few seconds, together with a basic dump of loadavg just to simplify searches. Save this as "psdump.sh" and make it executable with chmod +x psdump.sh.

#!/bin/bash

while true; do
    sleep 15
    date
    cat /proc/loadavg
    ps Swaux | awk '$3!=0.0 {print}'
done

You can run the script from a terminal so it saves to a file and goes in the background:

/home/lserni/psdump.sh > /home/lserni/psdump.log

Then, just wait. The file will display something like,

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  5.0  0.0 185412  5272 ?        Ss   Jan30 15340:14 /lib/systemd/systemd --system --deserialize 27
root        63  1.6  0.0      0     0 ?        S    Jan30 5042:59 [kswapd0]
root      1930  1.5  0.0  29008  2464 ?        Ss   Jan30 4816:50 /usr/sbin/cron -f
root      7446  0.1  0.0 3964564 60880 ?       Sl   Jan30 425:27 /opt/dell/srvadmin/sbin/dsm_om_connsvcd -run
mysql     7509  3.3 57.3 41322444 37757724 ?   Ssl  Aug17 599:39 /usr/sbin/mysqld
root      9634 17.1  4.9 6141588 3234672 ?     Ssl  Jan30 52154:34 /usr/local/bin/antani

The log file will report only processes with nonzero CPU percentage (children included). The columns you're interested in are probably %CPU and cumulated time (columns #3 and #10).

Answer 2

This should do it:

Top has a "batch" mode meant for noninteractive use accessible with -b. So this script records the top output every 10 seconds:

#! /bin/sh

set -u

recording=/tmp/top_recording.txt

top_interval_seconds=10

top -b -d "$top_interval_seconds" > "$recording"

Sadly, all processes/tasks except the first few are redundant, so I use Tr and Sed to view the report, leaving only the top three processes of each snapshot that Top produces:

#! /bin/sh

set -u

recording=/tmp/top_recording.txt

regexp_first_few_lines='top'\
'[^\x00]*\x00'\
'[^\x00]*\x00'\
'[^\x00]*\x00'\
'[^\x00]*\x00'\
'[^\x00]*\x00'\
'[^\x00]*\x00'\
'[^\x00]*\x00'\
'[^\x00]*\x00'\
'[^\x00]*\x00'\
'[^\x00]*\x00'

export LC_ALL=C

< "$recording" \
   tr '\n' '\0' | \
   sed -E -e 's/\x00top/\ntop/g' | \
   sed -E -e 's/^('"$regexp_first_few_lines"')(.|\x00)*$/\1/' | \
   tr '\0' '\n' | \
   less

To find high-CPU activity processes in Less, use:

/[0-9] [A-Za-z] ([^ ]| [^ ]|  [987])