0

When I do a software update (Ubuntu upgrade) and e.g. install new Linux kernels, sometimes I run out of space, which is a BAD THING. This is on Ubuntu Bionic 18.04, with systemd, so journald is installed.

Journald is configured by default to preserve 15% of the space on the drive for other processes (SystemKeepFree), and use a max of 10% of the space (SystemMaxUse). But it is often over those values.

So what triggers a vacuuming operation?

How can we get it to quickly clean up space when another operation needs it?

Is there a way to search journald for previous cleanup actions - are they logged?

Note that the grep capability is missing - why? (And see Ubuntu bug 1751657)

$ journalctl --grep journald
Compiled without pattern matching support
  • I do not know why it is exceeding SystemMaxUse=, sorry. But maybe you can quote some example numbers to illustrate it. The manpage for systemd 238 seems very clear about what it is claiming... and you are using a pretty recent version... it doesn't seem to have an excuse for overrunning SystemMaxUse=. – sourcejedi Oct 30 '18 at 17:18
1

I do not know why it is exceeding SystemMaxUse=, sorry.

EDIT: in comments, @nealmcb endorses the answer below, and suggests the journal was only exceeding SystemKeepFree=, and not exceeding SystemMaxUse=.

How can we get it to quickly clean up space when another operation needs it?

I don't think you can rely on that in general. SystemKeepFree= works in a less reliable way than SystemMaxUse=:

If the file system is nearly full and either SystemKeepFree= or RuntimeKeepFree= are violated when systemd-journald is started, the limit will be raised to the percentage that is actually free. This means that if there was enough free space before and journal files were created, and subsequently something else causes the file system to fill up, journald will stop using more space, but it will not be removing existing files to reduce the footprint again, either.

If you change the configuration file on a running system, you can cause it to be re-read by running systemctl restart systemd-journald

You can reduce space manually using one of the journalctl --vacuum... commands, such as journalctl --vacuum-size=500M.

It seems there is code to log cleanups, but in recent versions the message priority is set to "debug". Presumably this means the messages are suppressed by default. See: What do systemd "Vacuuming done, freed 0 bytes" messages mean?

  • I now think it is exceeding SystemMaxUse precisely because of the text you quoted, since my system became more than 15% free for other reasons (too much mail!) along the way. This should be more prominently highlighted where SystemMaxUse is described, and should also probably be fixed somehow if possible, since as it is there is a ratcheting effect over time, in the context of other services using disk space, which makes that option pretty useless, especially since it seems to only enforce it at boot time, which is every few weeks for me.... But thanks for the clarity and quick response! – nealmcb Nov 2 '18 at 15:11
  • @nealmcb I don't understand. For example, say you have a 10GB filesystem, and SystemMaxUse=2G. Then in my understanding, journald should never use more than 2G of the filesystem. If that causes you a problem, there are three things you might blame: 1) you set SystemMaxUse= too high 2) you need a bigger filesystem 3) the rest of the system is using too much space for some reason. I guess SystemKeepFree= is a heuristic to make problems less likely - especially since the journal takes more space than plain-text logfiles - but I haven't really worked out what the logic behind it is. – sourcejedi Nov 2 '18 at 19:22
  • 1
    Oops - sorry - I meant it is exceeding SystemKeepFree because of the text you quoted. I think that indeed means SystemKeepFree is of questionable utility, unless everyone using the disk implements something similar. – nealmcb Nov 3 '18 at 21:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.