How to analyze system errors when journalctl fails?

Question

One of my KVM servers (2 Xeon E5-2680 v2, 1 AMD Vega 10 GPU, Ubuntu 20.04.1 LTS) became unresponsive last night. Of the 5 VMs running on the server, only one could be reached. The server itself refused SSH connections, and I could not even get a screen over HDMI. I did not see any other solution than to reset it.

Having done that, I would like to better understand what was actually going on. The following journals are available on the system:

# journalctl --list-boots
-8 57c5ae37af1649379e82b349abb14f9d Sun 2020-05-24 20:25:57 CEST—Sun 2020-05-24 20:44:30 CEST
-7 c617acfdd3854669bd114d1d033cd5a7 Sun 2020-05-24 20:45:01 CEST—Mon 2020-05-25 19:21:48 CEST
-6 745df76c9d784907862118c7804a19ab Mon 2020-05-25 19:22:26 CEST—Mon 2020-05-25 19:42:17 CEST
-5 9781df6fa3494c4588d0cf4a99678e84 Mon 2020-05-25 19:42:59 CEST—Thu 2020-06-04 04:53:20 CEST
-4 db93d994719a4ee1ad8eb74932220898 Thu 2020-06-04 18:45:10 CEST—Thu 2020-06-04 19:16:38 CEST
-3 c6007ce834bd4933805138523549677e Thu 2020-06-04 19:17:20 CEST—Thu 2020-08-20 18:35:54 CEST
-2 c24b967697ce41a2ac6c1707936dc450 Thu 2020-08-20 18:36:23 CEST—Mon 2020-08-31 17:21:52 CEST
-1 b1efda1e7a3b42d4ae9a20f0c3b06fcf Mon 2020-09-07 09:49:24 CEST—Mon 2020-09-07 09:59:49 CEST
 0 f5de0a1534a7478e87847031156976d0 Mon 2020-09-07 10:00:19 CEST—Mon 2020-09-07 10:08:33 CEST

As you may already see from the list, the last 7 days are missing, I don't actually have access to a journal leading up to the system error.

Running journalctl --verify shows the following output.

1f23cc0: Invalid object                                                                                                                                                               
File corruption detected at /var/log/journal/f9decb319623482392299509c566049a/system@0005a744de309d9e-3dfdf0e20f2b37de.journal~:1f23cc0 (of 33554432 bytes, 97%).                     
FAIL: /var/log/journal/f9decb319623482392299509c566049a/system@0005a744de309d9e-3dfdf0e20f2b37de.journal~ (Bad message)
PASS: /var/log/journal/f9decb319623482392299509c566049a/system@24f780e4155245c0a176021b285d8b61-0000000000000001-0005a744de2ec87d.journal                                             
PASS: /var/log/journal/f9decb319623482392299509c566049a/user-1000@6da18e6709a04eb8809cb4af946ec557-00000000000008e3-0005a66900b2a7be.journal                                          
PASS: /var/log/journal/f9decb319623482392299509c566049a/system@24f780e4155245c0a176021b285d8b61-0000000000010c89-0005a8cfc4df2d27.journal                                             
PASS: /var/log/journal/f9decb319623482392299509c566049a/user-1000@6da18e6709a04eb8809cb4af946ec557-0000000000010c88-0005a8cfc4dec165.journal                                          
PASS: /var/log/journal/f9decb319623482392299509c566049a/system@24f780e4155245c0a176021b285d8b61-000000000001b654-0005ab341293df34.journal                                             
PASS: /var/log/journal/f9decb319623482392299509c566049a/user-1000@6da18e6709a04eb8809cb4af946ec557-000000000001bf95-0005ab56000dcbe1.journal                                          
PASS: /var/log/journal/f9decb319623482392299509c566049a/system@0005aeb4750f8660-d0ac81b12520109c.journal~                                                                             
PASS: /var/log/journal/f9decb319623482392299509c566049a/system@cf2a7210a86040e6aa7736d9b0a88e8b-0000000000000001-0005aeb4750dab5e.journal                                             
PASS: /var/log/journal/f9decb319623482392299509c566049a/system.journal                                                                                                                
PASS: /var/log/journal/f9decb319623482392299509c566049a/user-1000@6da18e6709a04eb8809cb4af946ec557-00000000000269b9-0005ad9c6a92da4a.journal                                          
PASS: /var/log/journal/f9decb319623482392299509c566049a/user-1000.journal  

Reading through some of the obvious web search results, it seems that it is currently not possible to repair a corrupted journal, it's just gone. Frankly, I find it a bit strange when the systemd lead writes that he does not see a need for corrupt journalctl entries to be fixed but maybe it's just me.

I'm really not sure what else to do. In my /var/log, I also have files called syslog but they also stop on August 31 and continue today. The same is true for kern. I looked through some other log files such as Xorg and dmesg. Honestly, I'm not even sure what to look for but nothing seems to jump at me.

Xorg.log shows only one error that seems unlikely to be the culprit for my problems:

[230912.637] (II) xfree86: Adding drm device (/dev/dri/card0)
[230912.637] (EE) /dev/dri/card0: failed to set DRM interface version 1.4: Permission denied

There seem to be no error or fail messages in dmesg.

I mean, everything works currently but this error seems to repeat itself every few weeks. Which other steps can I take to get a better understanding of this issue?

Answer 1

I don't think journalctl --list-boots shows all the journal files, just the boot events. From the manual:

--list-boots

Show a tabular list of boot numbers (relative to the current boot), their IDs, and the timestamps of the first and last message pertaining to the boot.

So it would seem there are no boots occurring between 2020-08-31 17:21:52 and 2020-09-07 09:49:24. If you know this to be false then there is indeed a gap in the journal.

But, we can check for all the (other) messages that happened since, or leading up, to a specific time. This specific event happened a long time ago as I write this, but it would be something like:

journalctl --since '2020-09-06' --until '2020-09-08

This will cover 48 hours between midnight the day before, until midnight the day after. You could filter it further with -k for only the kernel messages.

If there are logs, then this will print them. If they are indeed missing then we know that there must be some problem in the journal.

Journalctl is supposed to skip over corrupted messages in the journal, but still print what it can. I agree that the bug reports about this have been met with pretty flippant answers, but the intent is that bad messages should be skipped over. It looks from this --verify output that most of the log file is okay, with the corruption occuring towards the end (probably an incomplete message write, right at the point the system crashed).