22

I just noticed this verify option for journalctl and decided to give it a go, it's showing corruption, what might cause that? and what if anything should I do about it? should I investigate further?

journalctl --verify
PASS: /var/log/journal/19184893a1d645c7a43729e79b10a876/user-1000.journal
Invalid object contents at 3733856░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░   0%
File corruption detected at /var/log/journal/19184893a1d645c7a43729e79b10a876/system.journal:3733856 (of 91734016, 4%).
FAIL: /var/log/journal/19184893a1d645c7a43729e79b10a876/system.journal (Bad message)
Invalid object contents at 21575496░░░░░░░░░░░░░░░░░░░░░░░░░░░░░  45%
File corruption detected at /var/log/journal/19184893a1d645c7a43729e79b10a876/system@60e058db556e4de4b256d0b1ff176aa4-0000000000000001-0004e0b436d20aa1.journal:21575496 (of 44052480, 48%).
FAIL: /var/log/journal/19184893a1d645c7a43729e79b10a876/system@60e058db556e4de4b256d0b1ff176aa4-0000000000000001-0004e0b436d20aa1.journal (Bad message)
PASS: /var/log/journal/19184893a1d645c7a43729e79b10a876/user-1000@60e058db556e4de4b256d0b1ff176aa4-0000000000000a91-0004e0b4ff9a949a.journal
PASS: /var/log/journal/19184893a1d645c7a43729e79b10a876/user-1001.journal
14
+25

Currently, journalctl can detect corrupt logs but has no "fsck" type command to attempt repair. The journald will automatically switch to writing a new "clean" file as soon as it detects the problem, so theoretically data loss should be minimal.

Until there is a file-repairing command, finding the corrupt journal file and removing it is the only cure. You can find more on this in our Fedora mega-thread over making journal-only logging the default:

For tail corruptions your normal journalctl tool will provide you with as much information as is possible to salvage from the file. It will output the last complete log line and then finish. This is pretty close to how good you can get.

Things are different for corruptions in the middle. We have no nice tool for salvaging data from such corruption, but they could be written relatively easily. However, since they are highly unlikely due to the "append-only" model of the journal this hasn't been on our TODO list.

Of course, if you can identify what caused the problem initially and report it, that would be nice.

  • what is an FAA? – xenoterracide Aug 23 '13 at 4:40
  • @xenoterracide some sort of weird phone typo is what that is. :) – mattdm Aug 23 '13 at 12:09
  • 1
    @slm Yeah, there's a long bus ride from the office. :) Sorry for the time wasting -- hope you at least learned something about aviation. – mattdm Aug 23 '13 at 14:44
  • 1
    on my system at least corrupt journals seem to be common... is there any advice about how to identify what caused the problem? – xenoterracide Aug 23 '13 at 21:43
  • 2
    On every systemd system I have seen, corrupt journal logs are common. This answer is now almost 5 years old and as far as I can tell, there is no solution for this issue. – MountainX Mar 26 '18 at 19:37
2

It seems to be related to this thread on the ArchLinux wiki, titled: journalctl issues. It seems to have something to do with this setting SystemMaxUse in /etc/systemd/journald.conf.

The thread isn't conclusive but several people were having some luck in either clearing out the logs under /var/log/journal/* or increasing the value for SystemMaxUse.

  • unfortunately playing with that value doesn't seem to have helped. I'm suspect clearing logs helps because it deletes the invalid logs. – xenoterracide Aug 23 '13 at 0:14
  • @xenoterracide - I'll keep looking. Yeah cleaning out the logs seemed nonsensical to me too, but people were reporting it so I figured it was worth a shot. – slm Aug 23 '13 at 1:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.