We recently observed a high load average of about 1.5 on our embedded system, even though pretty much all processes are supposedly sleeping (according to
The system in question is a dual-core Cortex-A9 running a realtime Linux kernel (4.14.126) built using buildroot. We are using initramfs for our root filesystem and there is no swap, so there is definitely no disk I/O during normal operation.
After a bit of digging, we found out that the load is caused by a program called swupdate, which provides us with a convenient web interface for software updates (and we would very much like to continue using that).
When i use
time to estimate the average cpu-usage of that application (by calculating (user+sys)/real), i get a value of only about 1%, which doesn't make much sense considering the load average of 1.5.
I know that the load average also includes processes in the
TASK_UNINTERRUPTIBLE state, which don't contribute to cpu usage.
What i don't understand is why any of the threads/processes of that application would ever be in that state.
To further analyze the situation i have captured a kernel trace using lttng, which shows that the only thing swupdate does is this (every 50ms):
As you can see, there's a little bit of what appears to be socket-based IPC, and a select waiting for something.
In the IPC case, one thread appears to be mostly blocking in
nanosleep(), while the other is blocking in
accept(), neither of which should consume any system resources as far as i'm aware.
FYI: the time base for both screenshots is the same, and the IPC takes approx. 500-600µs in total (which, considering the interval of 50ms, fits quite nicely with the observed 1% CPU usage)
So, what is causing the load here?