One of my KVM servers (2 Xeon E5-2680 v2, 1 AMD Vega 10 GPU, Ubuntu 20.04.1 LTS) became unresponsive last night. Of the 5 VMs running on the server, only one could be reached. The server itself refused SSH connections, and I could not even get a screen over HDMI. I did not see any other solution than to reset it.
Having done that, I would like to better understand what was actually going on. The following journals are available on the system:
# journalctl --list-boots -8 57c5ae37af1649379e82b349abb14f9d Sun 2020-05-24 20:25:57 CEST—Sun 2020-05-24 20:44:30 CEST -7 c617acfdd3854669bd114d1d033cd5a7 Sun 2020-05-24 20:45:01 CEST—Mon 2020-05-25 19:21:48 CEST -6 745df76c9d784907862118c7804a19ab Mon 2020-05-25 19:22:26 CEST—Mon 2020-05-25 19:42:17 CEST -5 9781df6fa3494c4588d0cf4a99678e84 Mon 2020-05-25 19:42:59 CEST—Thu 2020-06-04 04:53:20 CEST -4 db93d994719a4ee1ad8eb74932220898 Thu 2020-06-04 18:45:10 CEST—Thu 2020-06-04 19:16:38 CEST -3 c6007ce834bd4933805138523549677e Thu 2020-06-04 19:17:20 CEST—Thu 2020-08-20 18:35:54 CEST -2 c24b967697ce41a2ac6c1707936dc450 Thu 2020-08-20 18:36:23 CEST—Mon 2020-08-31 17:21:52 CEST -1 b1efda1e7a3b42d4ae9a20f0c3b06fcf Mon 2020-09-07 09:49:24 CEST—Mon 2020-09-07 09:59:49 CEST 0 f5de0a1534a7478e87847031156976d0 Mon 2020-09-07 10:00:19 CEST—Mon 2020-09-07 10:08:33 CEST
As you may already see from the list, the last 7 days are missing, I don't actually have access to a journal leading up to the system error.
journalctl --verify shows the following output.
1f23cc0: Invalid object File corruption detected at /firstname.lastname@example.org~:1f23cc0 (of 33554432 bytes, 97%). FAIL: /email@example.com~ (Bad message) PASS: /firstname.lastname@example.org PASS: /email@example.com PASS: /firstname.lastname@example.org PASS: /email@example.com PASS: /firstname.lastname@example.org PASS: /email@example.com PASS: /firstname.lastname@example.org~ PASS: /email@example.com PASS: /var/log/journal/f9decb319623482392299509c566049a/system.journal PASS: /firstname.lastname@example.org PASS: /var/log/journal/f9decb319623482392299509c566049a/user-1000.journal
Reading through some of the obvious web search results, it seems that it is currently not possible to repair a corrupted journal, it's just gone. Frankly, I find it a bit strange when the systemd lead writes that he does not see a need for corrupt journalctl entries to be fixed but maybe it's just me.
I'm really not sure what else to do. In my
/var/log, I also have files called
syslog but they also stop on August 31 and continue today. The same is true for
kern. I looked through some other log files such as Xorg and dmesg. Honestly, I'm not even sure what to look for but nothing seems to jump at me.
Xorg.log shows only one error that seems unlikely to be the culprit for my problems:
[230912.637] (II) xfree86: Adding drm device (/dev/dri/card0) [230912.637] (EE) /dev/dri/card0: failed to set DRM interface version 1.4: Permission denied
There seem to be no error or fail messages in dmesg.
I mean, everything works currently but this error seems to repeat itself every few weeks. Which other steps can I take to get a better understanding of this issue?