Why load is high despite the fact that neither CPU or disk is overused

Question

I'm getting the following output from top:

Cpu(s): 43.8%us, 32.5%sy,  4.8%ni,  2.0%id, 15.6%wa,  0.2%hi,  1.2%si,  0.0%st
Mem:  16331504k total, 15759412k used,   572092k free,  4575980k buffers
Swap:  4194296k total,   260644k used,  3933652k free,  1588044k cached

the output from iostat -xk 6 shows the following:

Device: rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda       0.00   360.20   86.20  153.40  1133.60  2054.40    26.61     1.51    6.27   0.77  18.38
sdb       0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdd      22.60   198.80   17.40   31.60   265.60   921.60    48.46     0.18    3.70   1.67   8.20
sdc      16.80   218.20   22.20   23.40   261.60   966.40    53.86     0.21    4.56   1.49   6.78

Based on the above it looks like something must be overloaded. But what?

Questions

1. If its not the harddisk or the CPU then what?
2. It seems as though 15.6% of the CPU's time is spent waiting. What exactly could it be waiting for?

Answer 1

As a clarification point, load is not directly tied to CPU. This is one of the most common misconceptions about load. The fact that you mention disk seems to acknowledge that you're aware of this, but I just wanted to mention it as I see comments that indicate some believe otherwise.

Load is defined as the number of processes on the kernel run queue (meaning: waiting on system resources). This generally means processes waiting on CPU, disk, or network, but can be anything hardware really.
A "process" is not necessarily a full process either. A thread is defined as a "lightweight process", and each thread that is waiting increases the load count.

---

To figure out which processes are a problem:

Run top -H (the -H enables showing threads)

The keyboard shortcuts vary by version.

With newer top (3.3 and after):

Press f to bring up the field options.
Use the arrow keys to go to S = Process Status and press s.
Press q to go back to the main page.
Press Shift + R to reverse the sorting.

With older top (before 3.3):

Press Shift+o to bring up the sort options.
Then w to sort by process status.
Then Enter to go back to the main page.
Then Shift + R to reverse the sorting.

Then in the S column, look for processes which have D or R (they should now be at the top). These will be processes contributing to system load.

If the process shows a D, that means "uninterruptable sleep". Usually this is caused when the process is waiting on I/O (disk, network, etc).
If the process shows a R, that means it's just doing normal computation.

---

To find more about what those processes are doing:

With newer top (3.3 and after):

Press f to bring up the field options.
Use the arrow keys to go to WCHAN = Sleeping in Function and press d to enable it.
Then q to get back to the main page.

With older top (before 3.3):

Press f then y to enable the WCHAN field.

If your system has the necessary kernel options, and the wchan file is present on your system (I forget where it is and what it's called), the WCHAN field should show you what kernel function the process is currently running (if the field just shows a - or a ? on everything, you don't have support).
A bit of google here and you should be on your way.

If you don't have wchan support, you can always try an strace on the processes to find out what they're doing, but that's the difficult way.

Answer 2

Processes of short lifetime like compiling jobs or failing processes in a loop are often not visible in monitoring tools like top or iostat and so on.

In such cases the Linux Audit Framework will help

The culprit, a failure loop for example

while :; do gcc /dev/zero ; done >/dev/null 2>&1

To use auditd / auditctl:

apt-get install auditd
auditctl -a task,always
ausearch -i -sc execve

stolen from log all process launches

Answer 3

I had situation when NFS mounts disconnected and unfortunately I made a mistake and didn't use soft mount option, thus lot of processes staled on my Linux server, including monitoring, lsof, and even bash sessions....

After unmounting broken mounts, the system looked overloaded:

top - 00:03:48 up 15 days, 14:56,  3 users,  load average: 29, 21, 20

This looked terrible, but CPU usage under 15% and there are no disk I/O. I got some advices to go through ps, but this didn't help as it looked that processes are mostly sleeping.

Then man ps saved my night for sleep, and after investigation I found very important STATUS flags to look at, as later identified they were stuck processes.

Execute:

ps -e v

and look for processes which have D or SL in the STAT column. These were like zombie processes but not identified as Z -zombies.

D - means mostly disk (I/O) activity, but also if you run ps -e v few times and also iostat 3 and see no activity, this indicated that this is stuck i/o.

SL - this means there are Locked paged in memory of that process, thus if you can identify that this process should not behave like this, it's next possible candidate if it sticks for a longer period without change.

After investigation I then killed one by one, and my system load average became normal.