1

kworker process consumes 75% of one CPU. The offending kworker thread is related with ACPI stuff:

sudo cat /proc/THE_PID_OF_KWORKER_PROCESS/stack
[<ffffffff85c0c705>] acpi_ns_evaluate+0x1bc/0x23a
[<ffffffff85bffe09>] acpi_ev_asynch_execute_gpe_method+0x98/0xff
[<ffffffff85be4e30>] acpi_os_execute_deferred+0x10/0x20
[<ffffffff8588dc21>] process_one_work+0x181/0x370
[<ffffffff8588de5d>] worker_thread+0x4d/0x3a0
[<ffffffff85893f1c>] kthread+0xfc/0x130
[<ffffffff8588de10>] process_one_work+0x370/0x370
[<ffffffff85893e20>] kthread_create_on_node+0x70/0x70
[<ffffffff858791ba>] do_group_exit+0x3a/0xa0
[<ffffffff85e6a2b5>] ret_from_fork+0x25/0x30
[<ffffffffffffffff>] 0xffffffffffffffff

so I started debugging by rebooting with some acpi related kernel parameters, such as:

acpi=off : Completely solves the high cpu usage, but computer no longer suspends. 
acpi=ht : no effect, still high cpu usage
pci=noacpi : not booting at all
pnpacpi=off : no effect, still high cpu usage
noapic : worse, 100% cpu usage 
nolapic : worse, 100% cpu usage 

uname -a: Linux 4.13.0-1-amd64 #1 SMP Debian 4.13.4-1 (2017-10-01) x86_64 GNU/Linux

My root folder disk layout is: BTRFS over LVM over LUKS.

How can I find the root of the problem?

Update

I wasn't using my external hard drive which uses a DVD enclosure to be attached to the laptop.

enter image description here Today I reconnected the drive and kworker consumed that excessive amount of CPU again. Note that I didn't mount partition from the external drive, only attaching caused that cpu usage.

  • using any encryption ? – user1133275 Oct 17 '17 at 2:07
  • Yes. My system is running on a BTRFS subvolume on a BTRFS filesystem on an LVM volume on a LUKS partition. – ceremcem Oct 17 '17 at 6:03
  • That should go into the question. Don't comment, but do edit your question to improve it. Comments are for others. – Basile Starynkevitch Oct 17 '17 at 7:09
3

When I checked ACPI interrupts, I noticed that gpe6F had a very high trigger count:

root@HOST:~# grep . -r /sys/firmware/acpi/interrupts/

/sys/firmware/acpi/interrupts/ff_gbl_lock: 0  EN     enabled      unmasked
/sys/firmware/acpi/interrupts/gpe15:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe4F:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe43:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe7D:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe71:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe05:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe3F:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe33:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe6D:       0         disabled     unmasked
/sys/firmware/acpi/interrupts/gpe61:       0  EN     enabled      unmasked
/sys/firmware/acpi/interrupts/gpe2F:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe23:       0  EN     enabled      unmasked
/sys/firmware/acpi/interrupts/gpe5D:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe51:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe1F:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe13:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe4D:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe41:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe7B:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe0F:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe03:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe3D:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe31:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe6B:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe2D:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe21:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe5B:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe1D:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe78:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe11:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe4B:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/ff_pwr_btn:  0  EN     enabled      unmasked
/sys/firmware/acpi/interrupts/ff_slp_btn:  0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe0D:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe68:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe01:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/ff_pmtimer:  0     STS invalid      unmasked
/sys/firmware/acpi/interrupts/gpe3B:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe58:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe2B:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe48:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe1B:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe76:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe38:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe0B:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe66:       4  EN     enabled      unmasked
/sys/firmware/acpi/interrupts/gpe28:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe56:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe18:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe46:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe74:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe08:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/sci:  819678
/sys/firmware/acpi/interrupts/gpe36:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe64:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe26:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/error:       0
/sys/firmware/acpi/interrupts/gpe54:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe16:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe44:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe7E:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe72:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe06:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/ff_rt_clk:             disabled     unmasked
/sys/firmware/acpi/interrupts/gpe34:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe6E:  262969  EN     enabled      unmasked
/sys/firmware/acpi/interrupts/gpe62:       0  EN     enabled      unmasked
/sys/firmware/acpi/interrupts/gpe24:       0  EN     enabled      unmasked
/sys/firmware/acpi/interrupts/gpe5E:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe52:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe14:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe4E:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe42:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe7C:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe70:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe04:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe3E:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe32:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe6C:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe60:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe2E:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe22:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe5C:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe50:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe1E:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe79:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe12:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe4C:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe40:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe7A:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe0E:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe69:       0         disabled     unmasked
/sys/firmware/acpi/interrupts/gpe02:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe3C:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe30:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe6A:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe59:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe2C:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe20:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe5A:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe49:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe1C:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe77:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe10:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe4A:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe39:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe0C:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe67:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe00:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe3A:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe29:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe57:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe2A:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe19:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe47:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe1A:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe75:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe09:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe37:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe0A:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe65:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe27:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe55:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe17:       0     STS invalid      unmasked
/sys/firmware/acpi/interrupts/gpe45:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe7F:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/sci_not:     101
/sys/firmware/acpi/interrupts/gpe73:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe07:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe35:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe6F:  560719     STS enabled      unmasked
/sys/firmware/acpi/interrupts/gpe63:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe25:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe5F:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe_all:  823692
/sys/firmware/acpi/interrupts/gpe53:       0         invalid      unmasked

I disabled it:

root@HOST:~# echo "disable" > /sys/firmware/acpi/interrupts/gpe6F

and everything went back to normal:

Linux 4.9.0-6-amd64 (HOST)  05/01/2018  _x86_64_    (4 CPU)

12:30:27 PM     CPU     %user     %nice   %system   %iowait    %steal     %idle
12:30:30 PM     all      6.88      0.00      1.26      0.17      0.00     91.69
12:30:33 PM     all      6.45      0.00      1.17      0.17      0.00     92.20
12:30:36 PM     all      7.15      0.00      1.01      0.34      0.00     91.51
Average:        all      6.83      0.00      1.15      0.22      0.00     91.80
  • 3
    Could you please explain what you are doing here and why? Also it is difficult to see what of the information you have posted is important. – Raphael Ahrens May 1 '18 at 12:22
  • For me it was gpe61. I wish I knew what these were, but my idle CPU usage went down from 11-15% to around 1%. – TheWanderer Jun 16 '18 at 4:40
  • Thanks! This solved the issue for me. Should be accepted as best answer – Kyrol Jun 10 at 9:33
1

kworker is used by encryption (BTRFS) so you will see it consume CPU on disk IO (which can be background FS maintenance). You should run :

cryptsetup benchmark

to see if you are missing hardware acceleration.

  • How can I interpret the results? – ceremcem Oct 17 '17 at 22:12
  • @ceremcem a small aes-xts 512b comparison : forum.odroid.com/… if your numbers for that sub test are in the low end you can expect slowness/high cpu use. – user1133275 Oct 18 '17 at 6:22
  • The greatest score in the list is 1393 MB/s and mine is 1494 MB/s. So this is not the encryption case I suppose. – ceremcem Oct 18 '17 at 7:44
  • @ceremcem it still might be depending on IO, you could unmount all but a small partition, stop all services, to see if the issue persists.... or start checking logs maybe 'iotop -o -b -n 1' – user1133275 Oct 18 '17 at 7:55
  • Coincidentally I needed to remove my expansion hard disk from the laptop which was recently installed in place of DVD drive and installed the DVD drive temporarily. Now there is no kworker consuming that CPU resource. I'll try somethings more and let you know the results. – ceremcem Nov 1 '17 at 10:43
0

Apparently it was a hardware related problem. The kworker process was related with the external hdd enclosure where the enclosure had a switch to use for different brands:

enter image description here

Toggling the switch solved the issue.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.