I have been trying to debug random system freezes when running the 4.14.93-rt kernel. To this end, I have enabled the lockup detector in the kernel using the following config:
CONFIG_HAVE_HARDLOCKUP_DETECTOR_PERF=y CONFIG_LOCKUP_DETECTOR=y CONFIG_SOFTLOCKUP_DETECTOR=y CONFIG_HARDLOCKUP_DETECTOR_PERF=y CONFIG_HARDLOCKUP_CHECK_TIMESTAMP=y CONFIG_HARDLOCKUP_DETECTOR=y CONFIG_BOOTPARAM_HARDLOCKUP_PANIC=y CONFIG_BOOTPARAM_HARDLOCKUP_PANIC_VALUE=1 CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC=y CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC_VALUE=1
The goal is to trigger a kernel panic when a lockup occurs. I also enabled the NMI watchdog on the kernel cmdline:
Making use of kdump/kexec tools, I have configured the system to generate a kernel crash dump on kernel panic. The mechanism works when triggering a panic manually using:
echo c > /proc/sysrq-trigger
I can confirm that the system loads a dump-capture kernel in that case. However, when experiencing an actual lockup, the system just reboots when the watchdog kicks in. AFAIK there is no kernel panic occuring. There is no switch to the dump-capture kernel. No core dump, nothing stored in the logs.
Note that I enabled all relevant sysctl options:
kernel.panic = 1 kernel.panic_on_oops = 1 kernel.unknown_nmi_panic = 1 kernel.panic_on_unrecovered_nmi = 1 kernel.panic_on_io_nmi = 1 kernel.softlockup_panic = 1 kernel.hung_task_panic = 1
I see this behavior when experiencing a real-life system freeze. It also occurs when running a CPU hogging while-loop on all cores with high RT priority. I would expect this to be detected as a hung task and lead to a panic.
What could cause a reboot in this case without triggering the panic/kdump mechanism?