Analyse high load root cause

Question

This is a cents 5.7 32bit VM running inside a vmware 5.5. host. I see high values of load average with low CPU usage. The VM has 4 vCPU’s and load sometimes reach 20.

When I run vmstat I see high values in the 'r' column.

The question is how I find which process are inside the kernel run queue?.

I've tried what ever combintation of ps I've found in internet with no luck things like ps r -A

vmstat output:

[ ~]# vmstat 1 10
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 9  0      8 822516 322880 1593592    0    0     1    65    9    6  1  1 98  0  0
 7  0      8 823136 322880 1593584    0    0     0     0 9387 97411  8  9 84  0  0
53  0      8 823508 322880 1593588    0    0     0   236 8332 108913  9 12 79  0  0
64  0      8 818424 322888 1597548    0    0     0   116 9027 140988 10 11 79  0  0
69  0      8 820284 322888 1597548    0    0     0     0 9095 128715  8 10 83  0  0
64  0      8 820284 322888 1597692    0    0     0     0 8701 119305  9 11 80  0  0
 3  0      8 819540 322888 1597688    0    0     0  4704 9531 112734  8  8 84  0  0
81  0      8 818052 322888 1599452    0    0     0   224 8324 102409 10 13 77  0  0
 8  0      8 816192 322888 1601788    0    0     0  3240 9181 98478  9 11 80  0  0
 7  0      8 815076 322888 1601872    0    0     0     0 9250 104422 10  9 81  0  0 mpstat 1 10
06:04:03 PM  CPU    usr     nice    sys     iowait  irq     soft    steal   guest  idle
06:04:04 PM  all    9.32    0.00    8.82    0.00    0.25    4.03    0.00    0.00   77.58
06:04:05 PM  all    9.85    0.00    8.84    0.00    0.25    4.29    0.00    0.00   76.77
06:04:06 PM  all    8.29    0.00    5.78    0.00    0.50    4.77    0.00    0.00   80.65
06:04:07 PM  all    9.82    0.00    7.81    0.00    0.25    4.28    0.00    0.00   77.83
06:04:08 PM  all    8.84    0.00    5.30    0.00    0.25    4.29    0.00    0.00   81.31
06:04:09 PM  all   10.05    0.00    9.05    0.00    0.50    4.02    0.00    0.00   76.38
06:04:10 PM  all    9.60    0.00    7.32    0.00    0.51    4.04    0.00    0.00   78.54
06:04:11 PM  all    8.33    0.00    5.81    0.00    0.25    4.29    0.00    0.00   81.31
06:04:12 PM  all    9.57    0.00    7.05    0.00    0.25    4.03    0.00    0.00   79.09
06:04:13 PM  all    7.83    0.00    5.05    0.00    0.25    3.79    0.00    0.00   83.08
Average:     all    9.15    0.00    7.08    0.00    0.33    4.18    0.00    0.00   79.25

Answer 1

Usually load average is because of lots of things wanting CPU, but not always. A common culprit is a process waiting for IO - disk or network.

Try running ps -e v and looking for process state flags.

state    The state is given by a sequence of characters, for example, "RWNA". The      first character indicates the run state of the process:
D    Marks a process in disk (or other short term, uninterruptible) wait.
I    Marks a process that is idle (sleeping for longer than about 20 seconds).  
L    Marks a process that is waiting to acquire a lock.
R    Marks a runnable process.
S    Marks a process that is sleeping for less than about 20 seconds.
T    Marks a stopped process.
W    Marks an idle interrupt thread.
Z    Marks a dead process (a "zombie").

This is from the ps manpage, so you an find more detail there - R and D processes are probably of particular interest.

Answer 2

top is the best tool I know to find what process is using the system ressources.

on my machine :

$ top
top - 14:14:00 up 1 day,  2:00,  4 users,  load average: 0.24, 0.23, 0.24
Tasks: 235 total,   3 running, 232 sleeping,   0 stopped,   0 zombie
%Cpu(s):  3.2 us,  0.5 sy,  0.0 ni, 96.2 id,  0.1 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem:   7870416 total,  7097428 used,   772988 free,   346524 buffers
KiB Swap:  8081404 total,        0 used,  8081404 free.  3621000 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                
 4669 postgres  20   0 2037632  63924  37168 S  19.3  0.8   0:29.23 postgres                                                               
 4671 postgres  20   0 2037592  64476  38532 S  12.8  0.8   0:25.71 postgres                                                               
 4672 postgres  20   0 2037452  62992  38004 S  12.8  0.8   0:25.36 postgres                                                               
 1324 root      20   0  766268 212364 173952 S   6.4  2.7  22:34.20 Xorg                                                                   
 3804 ybounya   20   0  656468  23560  13244 S   6.4  0.3   1:44.77 gnome-terminal                                                         

So you can tell, currently PID 4669 (postgres) is using 19.3%, 4671 is using 12.6%, and so on. You can also see the memory and other key resources.