Can someone please explain the relation of CPU bottleneck on AIX regarding lcpu and r?

Question

# vmstat 60 2
System configuration: lcpu=12 mem=46080MB ent=1.10

kthr    memory              page              faults              cpu          
----- ----------- ------------------------ ------------ -----------------------
r  b   avm   fre  re  pi  po  fr   sr  cy  in   sy  cs us sy id wa    pc    ec
9  3 10845919 385420   0   0   0   0    0   0 5693 199747 54772 52 35  8  5  2.79 253.6
11  3 10855682 393324   0   0   0   0    0   0 6053 200390 51913 57 34  5  4  2.90 263.5

Although the average CPU usage was 95% (see "idle" column), the CPU isn't the bottleneck yet, since "r" was smaller then "lcpu".

http://aix4admins.blogspot.co.uk/2011/09/vmstat-t-5-3-shows-3-statistics-in-5.html

If runnable threads (r) divided by the number of CPU is greater than one -> possible CPU bottleneck (The (r) coulmn should be compared with number of CPUs (logical CPUs as in uptime) if we have enough CPUs or we have more threads.) High numbers in the blocked processes column (b) indicates slow disks. (r) should always be higher than (b); if it is not, it usually means you have a CPU bottleneck

Question: Can someone please explain this a little more detailed? Why can we say, that this system doesn't needs more CPU yet (r < lcpu)?

Answer 1

According to IBM documentation [1], the field r shows the number of runnable threads, precisely, the number of threads already running + number of threads waiting in a queue.

So, if r < lcpu, it means all the threads are on CPU and you have no threads waiting in a queue. In your case, 11 threads are running, and you even have 1 spare lcpu.

Let's say r = 20. In this case you are having CPU bottleneck, because there would be 12 threads on CPU + 8 threads waiting in queue for their turn to be scheduled. And CPU usage would be 100%.

[1] https://www.ibm.com/support/knowledgecenter/#!/ssw_aix_71/com.ibm.aix.cmds6/vmstat.htm

Answer 2

For reference, it's usually better to at least run vmstat -wtI 5 3 (-w gives wide output, -t gives timestamps so it's easier later to correlate your numbers to other performance monitoring results you may have running in parallel, and -I gives additional columns for file pagein/pageout) as you ran it for 60 seconds, and only looking at the vmstat output for the amount of time you're looking at is not really sufficient.

You should also always provide the output of lparstat -i for comparison and to put the information in context (capped/uncapped, dedicated CPU ...)