systemd-journal-flush.service Failed with result timeout - Failed to start Flush Journal to Persistent Storage

Question

I see this on my systemctl log, the systemd-journal-flush.service fails to start with a timeout error. Server is apparently running fine though. OS is CentOS 8. Logs don't tell much about the error. If anyone could help me I appreciate.

#sudo systemctl start systemd-journal-flush.service
Job for systemd-journal-flush.service failed because a timeout was exceeded.


#systemctl status systemd-journal-flush.service
● systemd-journal-flush.service - Flush Journal to Persistent Storage
   Loaded: loaded (/usr/lib/systemd/system/systemd-journal-flush.service; static; vendor preset: disabled)
   Active: failed (Result: timeout) since Sat 2023-03-18 04:08:07 GMT; 3min 28s ago
     Docs: man:systemd-journald.service(8)
           man:journald.conf(5)
  Process: 13447 ExecStart=/usr/bin/journalctl --flush (code=killed, signal=TERM)
 Main PID: 13447 (code=killed, signal=TERM)

Mar 18 04:06:37 systemd[1]: Starting Flush Journal to Persistent Storage...
Mar 18 04:08:07 systemd[1]: systemd-journal-flush.service: start operation timed out. Terminating.
Mar 18 04:08:07 systemd[1]: systemd-journal-flush.service: Main process exited, code=killed, status=15/TERM
Mar 18 04:08:07 systemd[1]: systemd-journal-flush.service: Failed with result 'timeout'.
Mar 18 04:08:07 systemd[1]: Failed to start Flush Journal to Persistent Storage.


#journalctl -xe
Mar 18 04:06:37 systemd[1]: Starting Flush Journal to Persistent Storage...
-- Subject: Unit systemd-journal-flush.service has begun start-up
-- Defined-By: systemd
-- Support: https://access.redhat.com/support
--
-- Unit systemd-journal-flush.service has begun starting up.
Mar 18 04:06:37 systemd-journald[860]: Runtime journal (/run/log/journal/9b0915b205da456da00e935c8704118b) is 14.3M, max 38.3M, 23.9M free.
-- Subject: Disk space used by the journal
-- Defined-By: systemd
-- Support: https://access.redhat.com/support
--
-- Runtime journal (/run/log/journal/9b0915b205da456da00e935c8704118b) is currently using 14.3M.
-- Maximum allowed usage is set to 38.3M.
-- Leaving at least 57.5M free (of currently available 368.6M of disk space).
-- Enforced usage limit is thus 38.3M, of which 23.9M are still available.
--
-- The limits controlling how much disk space is used by the journal may
-- be configured with SystemMaxUse=, SystemKeepFree=, SystemMaxFileSize=,
-- RuntimeMaxUse=, RuntimeKeepFree=, RuntimeMaxFileSize= settings in
-- /etc/systemd/journald.conf. See journald.conf(5) for details.
Mar 18 04:06:42 dbus-daemon[1068]: [system] Activating via systemd: service name='org.freedesktop.timedate1' unit='dbus-org.freedesktop.timedate1.service' requested by ':1.718' (uid=0 pid=10797 comm="cockpit-bridge --privileged " label="unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023")
Mar 18 04:06:42 systemd[1]: Starting System clock and RTC settings service...
-- Subject: Unit timedatex.service has begun start-up
-- Defined-By: systemd
-- Support: https://access.redhat.com/support
--
-- Unit timedatex.service has begun starting up.
Mar 18 04:06:42 dbus-daemon[1068]: [system] Successfully activated service 'org.freedesktop.timedate1'
Mar 18 04:06:42 systemd[1]: Started System clock and RTC settings service.
-- Subject: Unit timedatex.service has finished start-up
-- Defined-By: systemd
-- Support: https://access.redhat.com/support
--
-- Unit timedatex.service has finished starting up.
--
-- The start-up result is done.
Mar 18 04:08:07 systemd[1]: systemd-journal-flush.service: start operation timed out. Terminating.
Mar 18 04:08:07 systemd[1]: systemd-journal-flush.service: Main process exited, code=killed, status=15/TERM
Mar 18 04:08:07 systemd[1]: systemd-journal-flush.service: Failed with result 'timeout'.

Answer 1

The journald daemon is not flushing to /var/log/journal because that directory doesn't exist, and /etc/systemd/journald.conf does not tell it to create that directory. Unfortunately, journalctl --flush waits for the flush to complete, which it never will.

The solution which worked for me was this:

1. sudoedit /etc/systemd/journald.conf
2. After the [Journal] line, add a line that says Storage=persistent
3. Reboot.

If you read the journald.conf(5) man page, you'll see this explanation for how persistent differs from the default auto:

"auto" is similar to "persistent" but the directory /var/log/journal is not created if needed, so that its existence controls where log data goes.

By changing the configuration and rebooting, you're allowing journald to create the missing directory with the right permissions (which is, apparently, owner=root, group=systemd-journal, mode=02755, with an ACL granting explicit r-x access to groups adm and wheel). Of course, you could create it yourself and avoid the need to reboot.

Answer 2

This was caused by a bug fixed in systemd-239-75.el8.

Adding the Storage=persistent line to /etc/systemd/journald.conf as suggested in the other comment switches to persistent logging (it writes the journal to disk instead of using non-persistent ring buffer).