14

This is quite related to this question, but since it does not really have any satisfactory answers I figured I could ask a new question.

This screenshot shows htop indicating one core with 100% utilization, but with no process using any large amount of cpu:

htop with one core at 100% and process much lower

I assume this means that the kernel is using this much cpu for some unknown reason, but I haven't found a very good way of investigating this. (Looking into using eBPF for this now) I thought it might have something to do with my disk encryption and disk access, but iotop does not show any significant disk usage. I am running Arch Linux with a completely standard kernel.

The problem has appeared a couple of times lately and always goes away if I reboot, and always takes at least a couple of hours of on-time to appear.

Any ideas and suggestions for how to debug this or what the underlying cause could be would be very welcome.

Edit:

So this new screenshot shows htop set to display both kernel and user threads, but there is still no clear explanation for the high cpu usage:

htop with kernel and user threads

Edit 2:

Following screenshot shows results from bfptrace when running bpftrace -e 'profile:hz:99 /cpu == 0/ { @[kstack] = count(); }'. It seems that the kernel is spending a lot of time in acpi_os_execute_deferred for some reason.

enter image description here

16
  • 2
    Set htop to show all threads, including kernel threads (which it hides by default). Then you'll see what's causing it. Because it's either a process (whether in user or kernel mode at the time) or a true kernel thread (like a kworker). Nothing else would cause a high CPU usage to be reported there.
    – forest
    Commented Jun 7, 2021 at 22:42
  • 1
    @forest so I did this with shit-H and shift-k (see new screenshot), but there doesn't see too be any user or kernel threads that use any significant amount of cpu. Commented Jun 7, 2021 at 23:03
  • Oh, that is odd. The only thing I can think of now is something like a bug in system management mode, but I doubt that's it. One thing I'd do: try using cpuset to isolate your cores and prevent processes from being scheduled to CPU 0. Does it still use 100% in that situation?
    – forest
    Commented Jun 7, 2021 at 23:14
  • @forest so I tried this with sudo cset sheild --cpu=0 (I might have misunderstood how to use this command), however I got the message cset: **> 1 tasks are not movable, impossible to move and cset set -l lists a couple of tasks in the root set using cpus 0-7 and it is still at 100% Commented Jun 7, 2021 at 23:46
  • There's a kernel boot parameter which you can set in your bootloader which restricts the CPUs which is more effective than doing it at runtime since tasks don't need to be migrated. Since you say this happens even after you reboot (after some time), is it always the same CPU or does it vary?
    – forest
    Commented Jun 7, 2021 at 23:49

1 Answer 1

9

Finally found the answer. It turns out that the issue is the same as this question with additional information here and here. None of these mention the problem with htop showing zero usage though so that might be a unrelated problem.

As explained in the links above the answer was to use sudo grep . -r /sys/firmware/acpi/interrupts/ and then using echo "disable" /sys/firmware/acpi/interrupts/gpe6D to disable the problematic interupt (the one with the biggest number attached, in my case gpe6D).

To figure out that this was the problem I used bfptrace as explained in the question to do a kernel stack trace and figure out where the cpu was spending time and then bpftrace -e 'kprobe:acpi_ps_parse_aml /cpu == 0/ { printf("%d\n", tid); }' to find the kernel thread ID of one of the listed functions. Turns out the offending thread was kworker/0:3-kacpi_notify and then from some googling it turns out others have also had similar problems with this kernel thread.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .