5

I would like to have persistent logs via journald. I created /var/log/journal and reloaded the service. Now logs are saved on the disk. But the log size limit is low.

I added /etc/systemd/journald.conf. (Storage=persistent/auto doesn't matter, I tried both).

[Journal]
Storage=persistent
#Compress=yes
#Seal=yes
#SplitMode=uid
#SyncIntervalSec=5m
#RateLimitInterval=30s
#RateLimitBurst=1000
#SystemMaxUse=
SystemKeepFree=10G
SystemMaxFileSize=1G
#SystemMaxFiles=100
#RuntimeMaxUse=
#RuntimeKeepFree=
#RuntimeMaxFileSize=
#RuntimeMaxFiles=100
#MaxRetentionSec=
#MaxFileSec=1month
#ForwardToSyslog=yes
#ForwardToKMsg=no
#ForwardToConsole=no
#ForwardToWall=yes
#TTYPath=/dev/console
#MaxLevelStore=debug
#MaxLevelSyslog=debug
#MaxLevelKMsg=notice
#MaxLevelConsole=info
#MaxLevelWall=emerg

As you can see I only changed each journal file size to 1Gb and told that I want 10Gb free on the disk.

But journald tells me that he only has 4Gb log storage capacity.

$ sudo systemctl status systemd-journald
...
jan 20 15:44:26 host systemd-journald[1218]: System journal (/var/log/journal/) is 4.5G, max 4.0G, 0B free.
jan 20 15:44:26 host systemd-journald[1218]: Journal started

What did I miss?

$ systemctl --version
systemd 229
+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN
2

3 Answers 3

6

It's necessary to setup SystemMaxUse=100G e.g.

3
  • Is this true? can it really go beyond 4G?
    – nicocesar
    Apr 25, 2018 at 17:58
  • Seems to be able to go over 4G
    – ptman
    Aug 16, 2018 at 9:32
  • If you set a value with % (e.g. 12%), it is capped to 4G. If you set a value with G (e.g. 20G), it is respected.
    – Yajo
    May 3 at 8:51
1

From the man page:

SystemMaxUse=, SystemKeepFree=, SystemMaxFileSize=, SystemMaxFiles=, RuntimeMaxUse=, RuntimeKeepFree=, RuntimeMaxFileSize=, RuntimeMaxFiles=

[...]

The first pair defaults to 10% and the second to 15% of the size of the respective file system, but each value is capped to 4G. If the file system is nearly full and either SystemKeepFree= or RuntimeKeepFree= are violated when systemd-journald is started, the limit will be raised to the percentage that is actually free. This means that if there was enough free space before and journal files were created, and subsequently something else causes the file system to fill up, journald will stop using more space, but it will not be removing existing files to reduce the footprint again, either.

So yes, it maxes out at 4G, however, the following is interesting in your case:

journalctl and systemd-journald ignore all files with names not ending with ".journal" or ".journal~", so only such files, located in the appropriate directories, are taken into account when calculating current disk usage.

So, moving the log files regularly allows you to circumvent the limitation. Note that you will have to then ensure the logs do not fill up your file system somehow, removing old log files! You could, say have 1 file max 1G, then check regularly and move the file to ${filename}.n where n is 1-9, thus having max 10G ... provided you move the files when they reach 1G in size to the oldest ${filename}.n. To read the logs, you disregard the name, just oldest to youngest ...

2
  • Eventually the deal in SystemMaxUse=, thanks
    – 4xy
    Jan 20, 2018 at 19:32
  • 2
    I think if that is the case, then the man page is confusing.
    – thecarpy
    Jan 21, 2018 at 8:26
1

You may need to also raise SystemMaxFiles since systemd will start removing journal files from /var/log/journal/*/ folder before it reaches the System{MaxUse/KeepFree} limit because it's ran out of the number of journal files.

On my system, each file in /var/log/journal/*/systemd@*.journal was capped at 129Mb and with a default limit of only 100 SystemMaxFiles, the amount of logs stored was effectively capped at ~13Gb despite setting SystemMaxUse to 50G.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .