We have a curious situation regarding load average on our system. It's running an application called ZAG that is idle most of the day. But every 80 minutes or so, it has some sort of activity burst of 5-15 minutes' length. At the time of the burst, load average can climb to 60, 70, 80, 100 or more. What's interesting is this: during these high bursts, we'll see the CPU utilization percentages in htop show only 10-20% utilization per CPU. Furthermore, a script that I have written shows light CPU usage; during idle time this:

ps -eTo psr,user,pid,tid,cputime,class,rtprio,ni,pri,pcpu,stat,wchan:14,args | grep  ZAG | awk '{sum += $10} END{print sum;}'

returns perhaps 535.0... that is, adding up all the CPU percentages from my ZAG application return 535.0% of a CPU, or 5.35/32 or 16.7% utilization of all the CPUs on my system. So in short, not a single CPU is being driven anywhere close to 100% which we expect during mostly idle times.

During the incident, the result comes out to about 538.0 percent... just a little bump higher. I also see more threads on the run queue, as shown by

while true; do ps -eTo psr,user,pid,tid,cputime,class,rtprio,ni,pri,pcpu,stat,wchan:14,args | grep ZAG | grep ' Rl' | wc -l; sleep 0.5; done

So CPU utilization goes up, little, and there are more threads running. But not using much more CPU, it seems, even as the load average shoots up. There is consistently very little going on regarding disk i/o or network i/o; sar data shows no discernable increase during this burst. There is no increase in memory utilization, and the number of processes may increase by a couple out of some 1700 total processes on the system, but that's all. There is nothing in cron that takes place at these times. htop or top output shows that there is some certainly CPU utilization taking place at this time, mostly user CPU (less than 5% system CPU reported by top). So it doesn't look like anything is waiting on data.

I don't notice anything extraordinary in /proc/interrupts. Rescheduling interrupts seem to be heavy there, but I spot checked half a dozen cores, both even and odd NUMA nodes, and they appear to be stead at about 1400 per second per processor.

This is a 16-core machine with hyperthreads turned on (E5-2667 v4 processor). It has 36 ZAG processes and 729 ZAG therads, as show by ps -ef and ps -eTf, respectively.

So this makes me wonder: How come my CPU utilization percentages are so low, while my load average is so high? Is it because out of my 36 ZAG processes, I have over 700 threads, and that perhaps a thread that's in sched_yield() is still in the run queue, but not accumulating CPU? Or is sched_yield() no longer runnable, but in an noninterruptible state (see below)?

According to Brendan Gregg at https://www.brendangregg.com/blog/2017-08-08/linux-load-averages.html, "When load averages first appeared in Linux, they reflected CPU demand, as with other operating systems. But later on Linux changed them to include not only runnable tasks, but also tasks in the uninterruptible state (TASK_UNINTERRUPTIBLE or nr_uninterruptible). This state is used by code paths that want to avoid interruptions by signals, which includes tasks blocked on disk I/O and some locks. ... But don't Linux load averages sometimes go too high, more than can be explained by disk I/O? Yes, although my guess is that this is due to a new code path using TASK_UNINTERRUPTIBLE that didn't exist in 1993. In Linux 0.99.14, there were 13 codepaths that directly set TASK_UNINTERRUPTIBLE or TASK_SWAPPING (the swapping state was later removed from Linux). Nowadays, in Linux 4.12, there are nearly 400 codepaths that set TASK_UNINTERRUPTIBLE, including some lock primitives. It's possible that one of these codepaths should not be included in the load averages...."

1 Answer 1


I think I found it.

The question goes to this, quite simply:

How does one explain a high load average- that is, a lot of stuff on the run queue- and yet a low CPU utilization?

The answer, I believe, is in the sched_yield() system call. If the threads yield to other threads, they stay on the CPU's run queue but they may not be doing a whole lot.

See https://books.google.com/books?id=9yIEji1UheIC&pg=PA370&lpg=PA370&dq=sched_yield+reli[…]&hl=en&sa=X&ved=2ahUKEwimrNvwqrL1AhWIhOAKHQLyBDcQ6AF6BAgCEAM

In case that link disappears: This is "Understanding the Linux Kernel" by Bovet and Cesati, page 370 in the Second Edition.

It says, "The sched_ yield( ) system call allows a process to relinquish the CPU voluntarily without being suspended; the process remains in a task_running state, but the scheduler puts it at the end of the runqueue list. In this way, other processes that have the same dynamic priority have a chance to run. The call is used mainly by sched_fifo processes."

Here they mention "process", but in the man page at https://man7.org/linux/man-pages/man2/sched_yield.2.html the description says, "sched_yield() causes the calling thread to relinquish the CPU. The thread is moved to the end of the queue for its static priority and a new thread gets to run."

So conceptually we have the same behavior: The thing goes to the end of the runqueue list. Based on what we're seeing, it makes sense that the thread remains in the task_running state.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .