CPU load average + how to get cpu load average for history of X hours

Question

We all know we can get average CPU load as:

 uptime
 10:09:22 up 2 days,  1:44,  1 user,  load average: 20.01, 20.03, 22.05

but this shows only

load average over the last 1 minute is 22.05
load average over the last 5 minute is 20.03
load average over the last 15 minute is 20.01

We want a list of load averages for the last X hours. Is there any Linux (we're using RHEL) command that can show the history of average CPU load for the last X hours?

Answer 1

You would need additional software installed for that. You could use sar (see https://linux.die.net/man/1/sar ) or the monitoring-system of your choice.

sar -q will report load averages (among others...)

$ sar -q 1 5
Linux 4.9.0-9-amd64 (sds-ulm-edv-553-workstation)   09/10/2019  _x86_64_    (2 CPU)

02:04:43 PM   runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15   blocked
02:04:44 PM         0       158      0.00      0.02      0.10         0
02:04:45 PM         0       158      0.00      0.02      0.10         0
02:04:46 PM         0       158      0.00      0.02      0.10         0
02:04:47 PM         0       158      0.00      0.02      0.10         0
02:04:48 PM         0       158      0.00      0.02      0.10         0
Average:            0       158      0.00      0.02      0.10         0

The last line Average: is likely interesting here?

You can give a file to that: sar -q -f /var/log/sa/fileofyourchoice and then further process columns 4-6 of the output.

To get a quick overview of "what happened here?", you will likely need some graphical representation. Unfortunately I have no idea how to pipe sar-outout through GNUplot (or give it to grafana...) or the like to generate something useful.

Okay, this tickled me :-)

Produce a datafile...

LANG=C sar -q 1 250 | grep ':' | awk '{ print $1,$4,$5,$6 }' | sed '1d;$d' > datafile.txt

Find the maximal values:

$ datamash -t ' ' max 2 max 3 max 4 < datafile.txt 
2.53 1.55 1.1
$ head -1 datafile.txt | cut -d' ' -f1
20:20:53
$ tail -1 datafile.txt | cut -d' ' -f1
20:25:02

Now create a file plotting, where you will need those values for xrange and yrange:

$ cat plotting 
set title "Load over time"
set xdata time
set style data lines
set term png
set timefmt "%H:%M:%S"
set format x "%H:%M:%S"
set xlabel "Time"
set ylabel "Load"
set autoscale y
set xrange ['20:20:53':'20:25:02']
set yrange ['-0.01':'2.6']
set xtics rotate
set output "load_over_time.png"
plot "datafile.txt" using 1:2 t "loadavg-1" w lines, "datafile.txt" using 1:3 t "loadavg-5" w lines, "datafile.txt" using 1:4 t "loadavg-15" w lines

Now you can generate a graphic with $ gnuplot < plotting `

And now you have a graph in load_over_time.png:

If you have a monitoring-system (e.g. check_mk) in place, getting the history you need is much easier.