On a system with N CPUs, you can run N userspace threads using 100% of a CPU each. However when you try to use
ssh, the kernel would give
ssh a "fair" share of CPU time and allow you to log in.
When you see 100% CPU usage in VMware, and
ssh is not responding, you might have a busy-loop inside the kernel.
Make sure your server is not running a graphical interface on its local console. You want to be in text mode, so you can see any messages printed by the kernel. Now read:
The docs say the hard lockup detector, aka NMI watchdog, is enabled by default. Except it is disabled when the kernel runs in a VM. So in this case, the default is to only detect soft lockups.
* Hard lockup detection is enabled by default. Disable it, as guests
* can get false positives too easily, for example if the host is
-- arch/x86/kernel/kvm.c: kvm_guest_init()
I am confused by the rationale and history of this. I don't know why it considers the soft-lockup detector to be "safer" than the hard-lockup detector. Also the original change was justified with a different reason. "The guest PMU is still flushing out bugs. The idea is once the KVM PMU is stable enough, the default switches to hardlockup enabled by default". Lastly, it is mentioned that on some versions of hypervisors, it might not be possible to enable the NMI watchdog at all.
Assuming you are not massively over-committing CPUs in your hypervisor, you could see if it's possible to enable the NMI watchdog as well. You can use the
sysctl linked above, or the doc for the sysctl says you could also use the kernel boot option
Then test that you can see messages printed by the kernel:
Is the "local console" logged or at least persistent, so that you
would have noticed if a kernel crash was printed on it? It really
should be, but I am not sure familiar with how vSphere etc. would work
if you used an emulated serial console. If you are just using an
emulated video display, then it will be persistent already.
This VMWare article appears
to rely on the same assumption.
Make sure you have not disabled console logging. Run this
sudo sh -c "echo '<3>test' >/dev/kmsg"
It should show "test" on the console.
If this is an emulated video display, part of a crash message might scroll off the top of the screen. And if the kernel has crashed, then you cannot use shift+PageUp to scroll up. In principle, it can be more useful to have an emulated serial console which implements scrollback.
For kernel crashes, there are a couple of other crash dump suggestions in the VMWare link above.
-- Debian Stretch VM becomes quasi-unresponsive every few days
Most of the other instructions in the linked answer are about hung task messages. You will not necessarily see those if you have a lockup.
That said, it does also mention
sysrq+L might be an alternative way to get useful information, if you have a soft lockup. That will generate a kernel backtrace for each CPU. But the root cause might only be visible for one of the CPUs, so you really need to be able to capture quite a lot of messages there. It would be best if you have a serial console. If you have a video console, shift+PageUp might just work, assuming you don't have too many CPUs.