8

I've recently learned that journalctl is occupying a big chunk of my 16GB SD card (Raspberry Pi):

$ journalctl --disk-usage
Archived and active journals take up 312.1M in the file system.

I don't feel that journalctl and journald are pulling their weight in my use case for this machine. It's an old-ish RPi, and rsyslog is also running. I estimate my need for and use of journalctl is maybe "once in a blue moon". Consequently, I decided I would disable journald - which "feeds" journalctl. I assumed this would be straightforward, using systemctl to stop, or perhaps just disableing the systemd-journald.service so that it would not start on the next boot.

Before pulling the plug, I decided to do somee research. Instead of finding thousands of references that offered "how-to" advice, there were remarkably few results addressing my specific search term: "how to disable journald". Instead, the results mostly offered advice on reducing journald's resource consumption. I did find a couple of references that gave me pause:

An old thread in the ArchLinux forum suggested it was not possible to disable journald without repercussions; i.e. "Masking systemd-journald causes all kinds of dependency failures and drops you at an emergency prompt." But this post is now 10 years old...

The systemd-journald.service manual says, "stopping it [systemd-journald.service] is not recommended.". The documentation proceeds from there to discuss namespaces?

I've learned that the usual command to prevent systemd units from starting has no effect; i.e. it starts normally:

$ sudo systemctl disable systemd-journald.service
$ sudo reboot 

# ... and after boot & login:

$ systemctl status systemd-journald.service
● systemd-journald.service - Journal Service
     Loaded: loaded (/lib/systemd/system/systemd-journald.service; static)
     Active: active (running) since Fri 2022-06-03 07:30:29 UTC; 1min 59s ago
TriggeredBy: ● systemd-journald-audit.socket
             ● systemd-journald.socket
             ● systemd-journald-dev-log.socket
       Docs: man:systemd-journald.service(8)
             man:journald.conf(5)
   Main PID: 134 (systemd-journal)
     Status: "Processing requests..."
      Tasks: 1 (limit: 1598)
        CPU: 820ms
     CGroup: /system.slice/systemd-journald.service
             └─134 /lib/systemd/systemd-journald

...
$

How can journald be disabled? ... or can it be disabled?

If not, why would the systemd developers force this on users? (OK, yeah - asking for an opinion, so forget that part of the question.)

Improve this question
3
  • 1
    Please try disabling all the TriggeredBy's as well (systemd-journald-audit.socket systemd-journald.socket systemd-journald-dev-log.socket). This should work. I'm not going to test this on my working PC ;-)
    – Artem S. Tashkinov
    Jun 3, 2022 at 9:12
  • 1
    I'd like to configure a system (an RPi Zero) to be "read only" and minimize all CPU activity that's not necessary for what I'm doing. I'm here because I was thinking disabling system services, like logging, might be a good approach... at a minimum, an interesting exercise. Sounds like I might need to seek an embedded distro.
    – user154279
    Nov 29, 2022 at 21:47
  • 1
    @Seamus See my answer. I just spent half a day ripping systemd's benighted journaling from a bunch of systems because it couldn't handle the load of a system running with debug logs set. Loads that caused systemd to use 3+ entire CPUs while losing messages are now handled by a single rsyslogd process that takes about 17% of a single CPU. And it doesn't lose any log messages.
    – Andrew Henle
    May 26 at 16:37

3 Answers 3

Reset to default
5

The reason not to want to eliminate journald when you are using rsyslogd is that rsyslogd can get it's information from journald, and they play very well together this way.

That is sort of contra some of the Q&A's linked in comments here. Some observations about this on a current linux system (checked this on Fedora and Debian) which is configured to have journald use only volatile (in memory) storage, with rsyslog writing actual log files:

  • /var/run/log is a directory with one subdirectory, journal. This contains files written to by systemd-journald and read by rsyslogd. Note this directory is on a tmpfs partition, hence it counts as in memory volatile storage.
  • /dev/log is a symlink to /run/systemd/journal/dev-log.

The /etc/rsyslog.conf on these systems includes this (which is not the stock rsyslog.conf):

# Input
module (
        load="imjournal"
        ...
)

Fedora and Debian differ in the stock config here; the former does use imjournal as well as imuxsock (userspace socket).1 Debian uses imuxsock and imklog (kernel log), presumably because without imjournal it needs to listen to the kernel itself. I also presume this is a bit anachronistic.

Part of my point here is that completely ripping out journald may not be impossible but it is a bad idea. It is a core component of contemporary systemd based systems. It's deficit is that it potentially uses much more disk space than rsyslog, but this is easy to configure out by using Storage=volatile and (eg) RuntimeMaxUse=64M in /etc/systemd/journald,

Having rsyslogd fed from journald is a back-end/front-end relationship; rsyslog is great because it provides plain text logging as well as much more fine grained control over what gets logged where and how, while journald provides better integration with system services.

  1. The rsyslog recommendation is to use imuxsock as imuxjournal is less efficient and may suffer from other issues; see also Andrew Henle's answer about this and comments below.
Improve this answer
13
  • A well-reasoned argument - thank you. I feel like this is the correct answer, and the answer I expected as there's virtually nothing in the way of how-to-recipes explaining how journald might be removed or disabled. And since this is a comment, I'll express an opinion in closing: journald is to Linux what Siri is to Apple: You might not like it, but you're "forced" (more or less) to accept something you revile in order to get something you want or need.
    – Seamus
    Jun 5, 2022 at 20:32
  • 1
    Journald could certainly use some work on that front end part for contexts where it is the only logger (which it probably is now on most linux systems). It's only features are --unit to get service specific stuff, and of course it's colourful. BTW, I added a footnote that I realized might be important if you are putting this information into practice.
    – goldilocks
    Jun 5, 2022 at 21:36
  • my servers struggle with postfix on Debian stable with systemd: some messages that postfix logs correctly into journald do not get forwarded to the log-server via rsyslogd. with journalctl -feu "postfix*" they can be observed on the postfix-system while on log-server, tail -f /var/log/mail.info only provides some. I seems to me that on the postfix-server either rsyslogd or journald consider the sequences of log-messages to exhaust the burst-limits. Tests with either or both imuxsock as well as imjournal have not helped. Any hints?
    – luke
    Apr 9 at 7:32
  • @luke Have you tried tinkering with the rate limit? Imuxsock has a similar feature: rsyslog.com/doc/v8-stable/configuration/modules/… Although it is noted there that "the journal tends to drop messages when it becomes busy instead of forwarding them to the system log socket". I'd presume that doesn't apply to the journal, although maybe that is optimistic.
    – goldilocks
    Apr 10 at 15:30
  • they play very well together this way Not according to the rsyslog documentation: "Note that this module reads the journal database, what is considered a relatively performance-intense operation. As such, the performance of a configuration utilizing this module may be notably slower than when using imuxsock." and then (bolding is original to rsyslog): "Warning: Some versions of systemd journal have problems with database corruption,
    – Andrew Henle
    May 26 at 16:06
1

How can journald be disabled?

First, this is what the Rsyslogd documenation says about the imjournal module used to get log messages from systemd's journal (bolding is original, italics are my emphasis):

Note that this module reads the journal database, what is considered a relatively performance-intense operation. As such, the performance of a configuration utilizing this module may be notably slower than when using imuxsock. The journal provides imuxsock with a copy of all “classical” syslog messages, however, it does not provide structured data. Only if that structured data is needed, imjournal must be used. Otherwise, imjournal may simply be replaced by imuxsock, and we highly suggest doing so.

We suggest to check out our short presentation on rsyslog journal integration to learn more details of anticipated use cases.

Warning: Some versions of systemd journal have problems with database corruption, which leads to the journal to return the same data endlessly in a tight loop. This results in massive message duplication inside rsyslog probably resulting in a denial-of-service when the system resources get exhausted. This can be somewhat mitigated by using proper rate-limiters, but even then there are spikes of old data which are endlessly repeated. By default, ratelimiting is activated and permits to process 20,000 messages within 10 minutes, what should be well enough for most use cases. If insufficient, use the parameters described below to adjust the permitted volume. It is strongly recommended to use this plugin only if there is hard need to do so.

Do you have a "hard need" to feed "structured data" to rsyslog? And go a lot slower while using significantly more CPU cycles while doing that?

Replacing systemd's journaling with straight-to-rsyslog is easy:

  1. Edit /etc/rsyslog.conf. Remove all references to imjournal and imuxsock.

  2. Replace the above lines with

    module(load="imuxsock" # provides support for local system logging (e.g. via logger command)
    SysSock.Use="on") # Turn on message reception via local log socket;

  3. Run the following command:

    systemctl stop systemd-journal-flush.service systemd-journald.service systemd-journald-dev-log.socket systemd-journald.socket

  4. Run

    systemctl stop systemd-journal-flush.service systemd-journald.service systemd-journald-dev-log.socket systemd-journald.socket

  5. Run

    systemctl restart rsyslog

  6. Verify messages are appearing in /var/log/messages et al

You may need to tweak some of those commands - especially by disabling more systemd services.

Improve this answer
3
  • this is what the Rsyslogd documenation says about the imjournal module used to get log messages from systemd's journal -> No, that's a module used to do such, and the documentation you quote recommends instead the imuxsock module, which does not suffer from these problems and is much more the commonplace norm, I would guess (in fact I think it is the default on debian and derivations currently).
    – goldilocks
    May 26 at 17:10
  • @goldilocks And without that "structured data", what does the entire systemd/journald pipeline add to logging? While it sucks up stupendous amounts of CPU cycles to do a simple thing like log data? Pitch it into a black hole where it belongs, never to bother the rest of the universe again.
    – Andrew Henle
    May 26 at 17:14
  • If it isn't of interest or use to you, then you might as well not use it, of course. Conversely, though, I'd say on the vast majority of systemd based systems the journal doesn't "suck up stupendous amounts of CPU cycles", so most people might as well leave it be. The issue in the original question actually has to do with it being a storage pig, which is easy enough to limit in places where it becomes a problem.
    – goldilocks
    May 26 at 17:19
0

Disabling this service won't stop your device from functioning but logging will cease obviously. I see no other issues.

Improve this answer
11
  • 1
    "logging will cease obviously" All logging - or just journald logging?
    – Seamus
    Jun 3, 2022 at 6:45
  • In modern Linux distros most services log to the system journal (/dev/log). Some don't use it but they are rare. The kernel keeps its own log in a memory buffer (dmesg) but it's limited and gets rotated.
    – Artem S. Tashkinov
    Jun 3, 2022 at 6:46
  • Right - but what I'm asking is if rsyslog will continue "logging" as it does now. I don't want to stop rsyslog from logging - only journald because of its resource usage.
    – Seamus
    Jun 3, 2022 at 6:49
  • rsyslog itself is a logging server but if you had it installed with journald it doesn't log anywhere but redirects its messages to journald. You need to reconfigure it to make it actually log to /var/log.
    – Artem S. Tashkinov
    Jun 3, 2022 at 6:52
  • Please read this: unix.stackexchange.com/questions/205883/…
    – Artem S. Tashkinov
    Jun 3, 2022 at 6:55

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .