I am using Yocto to produce a custom image for a small embedded Linux system with SystemD Version 241. The root file system is Read-Only. I am using bind mounts and overlayfs to make the /var/log/journal directory exist on a separate Read/Write partition. I have a problem where systemd-journald gets "Amnesia" and does not remember previous boot logs, even though they are on the persistent Read/Write filesystem. This means journal cannot access or clean older log files from previous boots, even though the log files are present and valid on the filesystem.
# Setup overlayfs binds for various RW files VOLATILE_BINDS_append = " \ /persistent-storage/var/log /var/log\n\ "
The path /var/log exists:
root@me:/var/log# cd /var/log/ root@me:/var/log# ls -lrt total 9 drwxr-xr-x 2 root root 1024 Jun 3 01:50 nginx -rw-r--r-- 1 root root 5260 Jun 9 17:56 messages drwxr-sr-x 5 root systemd-journal 1024 Jun 9 18:00 journal root@me:/var/log# ls -lrt journal/ total 3 drwxr-sr-x 2 root systemd-journal 1024 Jun 9 17:56 5f6085cd81114e8688cf23b3bb91933e drwxr-sr-x 2 root systemd-journal 1024 Jun 9 17:57 de59603d1ea24e7582ed7d7ed3ac8fb0 drwxr-sr-x 2 root systemd-journal 1024 Jun 9 18:00 0c34cc794e6c4241a75774bbb8324102
I have a journald config file fragment in /lib/systemd/journald.conf.d/10-persistent-journal.conf that looks like this:
# By default the maximum use limit (SystemMaxUse) is 10% of the filesystem, and the minimum # free space (SystemKeepFree) value is 15% - though they are both capped at 4G. # The journals should be rotated automatically when they reach the SystemMaxFileSize value, # and the number of journals is controlled by SystemMaxFiles. If you prefer time based # rotation you can set a MaxFileSec to set the maximum time entries are stored in a single journal. [Journal] Storage=persistent SystemMaxFileSize=128M SystemMaxFiles=10 SystemMaxUse=256M SystemKeepFree=256M SyncIntervalSec=30
The problem is that even though I reboot several times, and journald successfully finds and writes logs to /var/log/journal, it can never find previous logs and has no knowledge about previous boot logs. This means I cannot vacuum previous logs and my partition runs out of space even though journald should maintain 50% of the partition free.
root@me:/# journalctl --list-boots 0 82fef865e29e481aae27bd247c10e591 Tue 2020-06-09 18:00:12 UTC—Tue 2020-06-09 18:15:23 UTC
root@me:/# ls -lrt /var/log/journal/ total 3 drwxr-sr-x 2 root systemd-journal 1024 Jun 9 17:56 5f6085cd81114e8688cf23b3bb91933e drwxr-sr-x 2 root systemd-journal 1024 Jun 9 17:57 de59603d1ea24e7582ed7d7ed3ac8fb0 drwxr-sr-x 2 root systemd-journal 1024 Jun 9 18:00 0c34cc794e6c4241a75774bbb8324102
Also, the following commands work:
root@me:/# journalctl -b 0 <information> root@me:/# journalctl -b 1 <information> root@me:/# journalctl -b 2 Data from the specified boot (+2) is not available: No such boot ID in journal
I read this post: Can be the journal path on a filesystem other than /?. And I tried the following mount file, however I see exactly the same behavior:
[Unit] Description=Persistent Journal Storage Bind [Mount] What=/anotherfs/journal Where=/var/log/journal Type=none Options=bind [Install] WantedBy=local-fs.target
What am I doing wrong and how can I get journald to work with persistent logs on a bind mount system?