We have a curious situation regarding load average on our system. It's running an application called ZAG that is idle most of the day. But every 80 minutes or so, it has some sort of activity burst of 5-15 minutes' length. At the time of the burst, load average can climb to 60, 70, 80, 100 or more. What's interesting is this: during these high bursts, we'll see the CPU utilization percentages in htop show only 10-20% utilization per CPU. Furthermore, a script that I have written shows light CPU usage; during idle time this:
ps -eTo psr,user,pid,tid,cputime,class,rtprio,ni,pri,pcpu,stat,wchan:14,args | grep ZAG | awk '{sum += $10} END{print sum;}'
returns perhaps 535.0... that is, adding up all the CPU percentages from my ZAG application return 535.0% of a CPU, or 5.35/32 or 16.7% utilization of all the CPUs on my system. So in short, not a single CPU is being driven anywhere close to 100% which we expect during mostly idle times.
During the incident, the result comes out to about 538.0 percent... just a little bump higher. I also see more threads on the run queue, as shown by
while true; do ps -eTo psr,user,pid,tid,cputime,class,rtprio,ni,pri,pcpu,stat,wchan:14,args | grep ZAG | grep ' Rl' | wc -l; sleep 0.5; done
So CPU utilization goes up, little, and there are more threads running. But not using much more CPU, it seems, even as the load average shoots up. There is consistently very little going on regarding disk i/o or network i/o; sar data shows no discernable increase during this burst. There is no increase in memory utilization, and the number of processes may increase by a couple out of some 1700 total processes on the system, but that's all. There is nothing in cron that takes place at these times. htop or top output shows that there is some certainly CPU utilization taking place at this time, mostly user CPU (less than 5% system CPU reported by top). So it doesn't look like anything is waiting on data.
I don't notice anything extraordinary in /proc/interrupts. Rescheduling interrupts seem to be heavy there, but I spot checked half a dozen cores, both even and odd NUMA nodes, and they appear to be stead at about 1400 per second per processor.
This is a 16-core machine with hyperthreads turned on (E5-2667 v4 processor). It has 36 ZAG processes and 729 ZAG therads, as show by ps -ef and ps -eTf, respectively.
So this makes me wonder: How come my CPU utilization percentages are so low, while my load average is so high? Is it because out of my 36 ZAG processes, I have over 700 threads, and that perhaps a thread that's in sched_yield() is still in the run queue, but not accumulating CPU? Or is sched_yield() no longer runnable, but in an noninterruptible state (see below)?
According to Brendan Gregg at https://www.brendangregg.com/blog/2017-08-08/linux-load-averages.html, "When load averages first appeared in Linux, they reflected CPU demand, as with other operating systems. But later on Linux changed them to include not only runnable tasks, but also tasks in the uninterruptible state (TASK_UNINTERRUPTIBLE or nr_uninterruptible). This state is used by code paths that want to avoid interruptions by signals, which includes tasks blocked on disk I/O and some locks. ... But don't Linux load averages sometimes go too high, more than can be explained by disk I/O? Yes, although my guess is that this is due to a new code path using TASK_UNINTERRUPTIBLE that didn't exist in 1993. In Linux 0.99.14, there were 13 codepaths that directly set TASK_UNINTERRUPTIBLE or TASK_SWAPPING (the swapping state was later removed from Linux). Nowadays, in Linux 4.12, there are nearly 400 codepaths that set TASK_UNINTERRUPTIBLE, including some lock primitives. It's possible that one of these codepaths should not be included in the load averages...."
