0

Our VPS SSD host based on KVM has experienced crashes during the last months: I was given just two screenshots of logs from my collegues and no journalctl log at all. We have two other hosts but from VPS provider based on VMWare: those always run fine, the three hosts running on KVM always gave troubles, crashing up to 5 times per month.

My team restored the crashed machine just by putting back a snapshot of another machine with the same CentOS image; they used this procedure other times and of course (?) other crashes could occur again in future.

As you can see in the first screenshot, a corruption of in-memory data was detected: I was given these two screenshots and the task to replicate the error in order to understand what happened. With no logs it might be quite of a challenge.

I know it's hard to understand without logs, but I collected a few infos:

  1. the crash occurred in not regular time intervals;
  2. the logs show a timestamp of three seconds and a half after the boot;
  3. this boot phase might involve initramfs.

I'm taking my best guess here: maybe someone installed something and rebooted the machine, and maybe update-initramfs went bad and corrupted something. Previous logs were lost; the VPS provider granted us their hardware is running fine and no instances of QEMU or KVM crashed...so, no idea what caused the file system corruption; it's useless to run xfs_repair if another crash will happen since all hosts share the same CentOS ISO.

I know this might seem a question belonging to ServerFault, but my question is more Linux related: in general, and not related to my particular issue, what could compromise initramfs?

Here's the screenshots:

First screenshot log

Second screenshot log

1

This worked for me:

Once you are are in emergency mode run

xfs_repair -L /dev/dm-0 && reboot

Log in and in a terminal run

sudo xfs_fsr /dev/mapper/centos-root sudo xfs_fsr /dev/dm-0 reboot

Not 100% sure what the issue is but all my centOS VMs regardless of version does this. I was working on this issue for a week.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.