systemd-journald persistent logs do not work with bind mount /var/log

Question

I am using Yocto to produce a custom image for a small embedded Linux system with SystemD Version 241. The root file system is Read-Only. I am using bind mounts and overlayfs to make the /var/log/journal directory exist on a separate Read/Write partition. I have a problem where systemd-journald gets "Amnesia" and does not remember previous boot logs, even though they are on the persistent Read/Write filesystem. This means journal cannot access or clean older log files from previous boots, even though the log files are present and valid on the filesystem.

Yocto volatile binds

# Setup overlayfs binds for various RW files
VOLATILE_BINDS_append = " \
    /persistent-storage/var/log /var/log\n\
"

The path /var/log exists:

root@me:/var/log# cd /var/log/
root@me:/var/log# ls -lrt
total 9
drwxr-xr-x 2 root root            1024 Jun  3 01:50 nginx
-rw-r--r-- 1 root root            5260 Jun  9 17:56 messages
drwxr-sr-x 5 root systemd-journal 1024 Jun  9 18:00 journal
root@me:/var/log# ls -lrt journal/
total 3
drwxr-sr-x 2 root systemd-journal 1024 Jun  9 17:56 5f6085cd81114e8688cf23b3bb91933e
drwxr-sr-x 2 root systemd-journal 1024 Jun  9 17:57 de59603d1ea24e7582ed7d7ed3ac8fb0
drwxr-sr-x 2 root systemd-journal 1024 Jun  9 18:00 0c34cc794e6c4241a75774bbb8324102

I have a journald config file fragment in /lib/systemd/journald.conf.d/10-persistent-journal.conf that looks like this:

# By default the maximum use limit (SystemMaxUse) is 10% of the filesystem, and the minimum
# free space (SystemKeepFree) value is 15% - though they are both capped at 4G.
# The journals should be rotated automatically when they reach the SystemMaxFileSize value,
# and the number of journals is controlled by SystemMaxFiles. If you prefer time based
# rotation you can set a MaxFileSec to set the maximum time entries are stored in a single journal.
[Journal]
Storage=persistent
SystemMaxFileSize=128M
SystemMaxFiles=10
SystemMaxUse=256M
SystemKeepFree=256M
SyncIntervalSec=30

The problem is that even though I reboot several times, and journald successfully finds and writes logs to /var/log/journal, it can never find previous logs and has no knowledge about previous boot logs. This means I cannot vacuum previous logs and my partition runs out of space even though journald should maintain 50% of the partition free.

root@me:/# journalctl --list-boots
0 82fef865e29e481aae27bd247c10e591 Tue 2020-06-09 18:00:12 UTC—Tue 2020-06-09
 18:15:23 UTC

Even though:

root@me:/# ls -lrt /var/log/journal/
total 3
drwxr-sr-x 2 root systemd-journal 1024 Jun  9 17:56 5f6085cd81114e8688cf23b3bb91933e
drwxr-sr-x 2 root systemd-journal 1024 Jun  9 17:57 de59603d1ea24e7582ed7d7ed3ac8fb0
drwxr-sr-x 2 root systemd-journal 1024 Jun  9 18:00 0c34cc794e6c4241a75774bbb8324102

Also, the following commands work:

root@me:/# journalctl -b 0
<information>
root@me:/# journalctl -b 1
<information>

root@me:/# journalctl -b 2
Data from the specified boot (+2) is not available: No such boot ID in journal

I read this post: Can be the journal path on a filesystem other than /?. And I tried the following mount file, however I see exactly the same behavior:

[Unit]
Description=Persistent Journal Storage Bind

[Mount]
What=/anotherfs/journal
Where=/var/log/journal
Type=none
Options=bind

[Install]
WantedBy=local-fs.target

What am I doing wrong and how can I get journald to work with persistent logs on a bind mount system?

Answer 1

I have been fighting the same problem for quite a while now. I have a Yocto-based distro where I have a read-only rootfs and the /var/log folder is mounted in a different partition using volatile-binds. Once I got journald to log to the new partition I noticed the same thing, journalctl --list-boots would only show the current boot and not the older ones. After a lot of trial and error I ended up figuring out why.

As you point out, /var/log/journal shows 3 folders:

root@mir-edb-intel-gen1:~# ls -lrt /var/log/journal/
total 24
drwxr-sr-x+ 2 root systemd-journal 4096 Mar  5 09:24 7f7f3d516b554d718d688a0b0fa9648e
drwxr-sr-x+ 2 root systemd-journal 4096 Mar  5 09:25 93eb74229e9244a4b7f60c90acacc12f
drwxr-sr-x+ 2 root systemd-journal 4096 Mar  5 10:57 4a9652b4b33a4a099bdc6be8c6fb2b1a

And journalctl --list-boots shows only 1 entry:

 0 6cb4f9f236954f69937be217f36c1ce2 Fri 2021-03-05 10:57:48 UTC—Fri 2021-03-05 11:01:22 UTC

However, if you run journalctl -D /var/log/journal --list-boots all the 3 boots appear as expected:

root@mir-edb-intel-gen1:~# journalctl -D /var/log/journal --list-boots
-2 9ab6d00dea9d41da9c76bf2a3f64895c Fri 2021-03-05 09:24:20 UTC—Fri 2021-03-05 09:25:03 UTC
-1 00509584245d44d599d88db8dccd4177 Fri 2021-03-05 09:25:37 UTC—Fri 2021-03-05 10:14:42 UTC
 0 6cb4f9f236954f69937be217f36c1ce2 Fri 2021-03-05 10:57:48 UTC—Fri 2021-03-05 11:01:22 UTC

Although I'm not that knowledgeable in systemd, I believe that the problem is due to the fact that we are flushing the journal from /run/log/journal to /var/log/journal every time we boot instead of writing more to the existing log in /var/log/journal. Then, when you call journalctl it only looks at the newest folder, therefore being unable to find the previous boots unless you specifically tell it to look at the whole /var/log/journal.

Edit: @Jaap Joris Vens seems to have a good explanation for the behaviour in this answer

Edit2: In our case the /etc/machine-id file is an empty file as explained in the poky/meta/classes/rootfs-postcommands.bbclass:

#
# A hook function to support read-only-rootfs IMAGE_FEATURES
#
read_only_rootfs_hook () {

    [...]

    if ${@bb.utils.contains("DISTRO_FEATURES", "systemd", "true", "false", d)}; then
    # Create machine-id
    # 20:12 < mezcalero> koen: you have three options: a) run systemd-machine-id-setup at install time, b) have / read-only and an empty file there (for stateless) and c) boot with / writable
        touch ${IMAGE_ROOTFS}${sysconfdir}/machine-id
    fi
}

Answer 2

Systemd-Journald start at very early stage of the system boot your mount ( /var/log) probably happen after the service is started /var/log. more detail on how you mount your RW partition is needed.

You can try to restart the jounrald service after the system boot.

Storage= Controls where to store journal data. One of "volatile", "persistent", "auto" and "none". If "volatile", journal log data will be stored only in memory, i.e. below the /run/log/journal hierarchy (which is created if needed). If "persistent", data will be stored preferably on disk, i.e. below the /var/log/journal hierarchy (which is created if needed), with a fallback to /run/log/journal (which is created if needed), during early boot and if the disk is not writable. "auto" is similar to "persistent" but the directory /var/log/journal is not created if needed, so that its existence controls where log data goes. "none" turns off all storage, all log data received will be dropped. Forwarding to other targets, such as the console, the kernel log buffer, or a syslog socket will still work however. Defaults to "auto" in the default journal namespace, and "persistent" in all others.

Additional info available here: link-1, link-2, link-3, link-3, link-5, link-6, link-7

Answer 3

A possible workaround would be a symlink from /var/log/journal to the other partition.

On my read-only-rootfs Yocto system I setup this symlink in one of my systemd services that starts the application. Until the symlink is in place all log goes into RAM, and is later flushed to the symlinked partition.

mkdir -p /data/log/journal
systemd-tmpfiles --create --prefix /data/log/journal
ln -s /data/log/journal /var/log/journal
journalctl --flush

Answer 4

Does journald still write to /run/log/journal/ even after /var/log/journal/ has been created? If so it could still be reading from the volatile /run/ directory.

Did you change another parameter between running # journalctl --list-boots and # journalctl -b 1? Because it seems to be reading from a file that does not exist judging by your output. Whatever configuration you had when you ran # journalctl -b 1 seems to be keeping persistence.

Apologies, I thought this would be more appropriate as a comment than an answer but I don't have enough reputation to comment yet.