A Lenovo Thinkpad X1 Carbon running Arch Linux updated its packages on December 15th since October 22nd. After a reboot the LED indicators of the Mute Mic and Mute Speakers keys stopped working.
I started my investigation by simply searching for phrases like "Mute LED not working" but couldn't find anything recent. I knew it had to be recent since the symptoms emerged exactly after an update.
Other buttons worked fine, Caps Lock, Fn Lock, but only the Mute buttons didn't work.
Finding the responsible package
/var/log/pacman.log/ I checked the packages that were updated. It were a whole lot, but nothing that could interfere with my Thinkpad buttons, except for an update to PulseAudio, ALSA, and the Linux Kernel.
I decided I would check the kernel first. To downgrade the kernel to the previous version I executed
~ # pacman -U /var/cache/pacman/linux-4.18.16.arch1-1-x86_64.pkg.tar.xz
After rebooting the lights worked again. I now knew for certain that the problem came from within an update to the Linux kernel.
Finding the responsible kernel version
I now know Kernel 4.18.16 is working, and I know 4.19.8 is not!
From the Arch Linux Package Archive (https://archive.archlinux.org/) I downloaded version 4.19 through 4.19.8, knowing the kernel broke somewhere in between.
Because I was on 4.18.16, I upgraded to 4.19.4. 19.4 worked like 18.16 did, so the bug was introduced after 19.4 but before 19.8
Next up was 19.6. This version also works fine, so I now know the bug was introduced in 19.7. After upgrading once more, sure enough, 4.19.7 was the first release where this "regression" (as they call it), took place.
Finding the commit responsible
Thanks to linux being open source, you can look up the changelog of every linux release on https://kernel.org
Here is the changelog for version 4.19.7: https://cdn.kernel.org/pub/linux/kernel/v4.x/ChangeLog-4.19.9
Warning, it is very big! To find some indication of where the problem has begun, I decided to CTRL+F some keywords into the file. First I tried "led", but there were no commits that looked promising. Then I searched for "mute", but again, no hits.
After a few other keywords I tried "carbon", and I found a commit named:
ALSA: hda/realtek - fix the pop noise on headphone for lenovo laptops
My first thought was "Oh great, they fixed that!", but since this was also the only commit mentioning Lenovo, and this was the changelog of the regressed kernel, my best bet was to investigate.
Thankfully this commit included a BugLink to launchpad.net: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1805079
I thought that there I could ask the developer if they too had a problem with their LED lights. But I didn't even need to ask, since another user already described the problem I was having: (Link)
I'm on Lenovo ThinkPad X1 Carbon 6th, model 20KH006JGE.
After upgrading to Linux 4.19.7, the audio mute and
microphone mute LEDs (on F1 and F4 keys) stopped working.
The creator of the commit had already responded, and had even provided a solution! Great stuff, but now what? I don't want to compile my own kernel... Yuck! I need to know when this fix will be implemented.
Finding the repairing commit
Luckily the whole linux kernel is on GitHub! https://github.com/torvalds/linux
I dove into the commit history, and CTRL+F'd for "LED", and sure enough, on page 4 I found the fixing commit! (link)
ALSA: hda/realtek - Fix the mute LED regresion on Lenovo X1 Carbon
Awesome, it seems like this commit is already posted and reviewed, so where is it now? When is the fix going to be released?
Finding out when the fix will be released
So linux' releases are maintained by Greg Kroah-Hartman. You might have seen his name on top of the changelog we looked at earlier. Every few days he gathers usefull commits from the repository and bundles them into a new stable.
You can track the progress and discussion of the release cycle in the mailing list stable from kernel.org. If you don't want to subscribe, but just read, you can find an archive right here: https://www.spinics.net/lists/stable/
There I simply pulled out my trusty CTRL+F once again and searched for "LED", and hell yeah, sure enough: [PATCH 4.19 140/142] was the commit I was looking for.
Greg has included the commit we wanted and is currently reviewing it's release. It is only a couple of days before I closes discussion, followed by release of 4.19.10.
Finally: The solution
To fix the issue, please downgrade the kernel to at most version 4.19.6.
You can also wait a couple of days, since version 4.19.10 will fix the regression, and it is expected to be released on December 16th or 17th.
I hope my journey was mildly interesting to read, and can help you troubleshoot your issues in the future. I learned about changelogs, commits, repo's, releases, mailing lists and a whole lot more, so I just had to share.
Kind regards and have fun!