1

I have an embedded device that has both systemd and rsyslog. journald is the main mechanism for local logging and rsyslog for remote logging. rsyslog isn't always running and is started only when debugging is needed.

Once rsyslog is started it forwards only the events that happen from that point in time. I'm trying to get it to forward all the logs that are available in journald at the moment.

The proper way to do this would be to enable $ModLoad imjournal in /etc/rsyslog.conf. This would allow rsyslog to read journald's archived files.

I'm on an embedded system which doesn't have imjournal module available for rsyslog and am looking for a workaround.

One such workaround would be to have journald resend all the archived logs from before to the socket (now active) rsyslog is listening to.

Is there a way to do this?

1

Such a feature is not really available in journald.

The journald.conf man page documents this limitation of ForwardToSyslog=:

If forwarding to syslog is enabled but nothing reads messages from the socket, forwarding to syslog has no effect.

Which is why messages from early boot are lost. The man page also states:

With [imjournal], messages do not have to be read immediately, which allows a logging daemon which is only started late in boot to access all messages since the start of the system.

Which is why using such a method is recommended.

I guess a "kludge" could be done when rsyslog is first started, by feeding it messages stored in the journal since boot, using journalctl to read them and logger to feed them to rsyslog. Perhaps something as simple as this could work:

journalctl -b | logger -u /run/systemd/journal/syslog

Though it's possible logger will try to add dates and tags to the messages and the final effect might look different from the forwarded messages... Perhaps you might need something more raw or low-level than logger, or maybe reformat journalctl output so it's in a format rsyslog will understand...

This solution has a race condition, since it's quite possible some messages will be lost (or duplicated) in between this initial feed and journald's forwarding.

Getting imjournal working is definitely a better solution. I'm curious as to why it can't be made available, after all both systemd and rsyslog software were compiled and built for your device, so it would be at least technically possible to build the imjournal mode as well... Perhaps there's some cross-compiling involved which might pose challenges linking to systemd libraries, but I'm sure that's a solvable problem, so maybe consider asking a question towards getting that working instead.


Another possible solution to consider is using the native journal remote protocols (rather than the syslog protocol and daemon) to centralize your logs.

There's systemd-journal-remote which you can run in "sink" mode on a remote host to receive entries from your embedded device, to save them locally. And systemd-journal-upload, which you can run on your embedded device to push journal data to the remote host.

This should both preserve messages from early boot and also keep all the metadata, since the messages do not have to be converted to the syslog format. It also has the advantage that you don't need to keep a locally running rsyslog daemon on your embedded device.

(The journal also supports a "pull" model, where you run systemd-journal-gatewayd on your embedded device and configure systemd-journal-remote to pull from it.)

  • Tried a similar workaround with systemd-cat however it masks what application is logging. It can be done, I'm not sure if it's an effort worth doing because of the description for imjournal. There is a line there also stating: The journal provides imuxsock with a copy of all “classical” syslog messages, however, it does not provide structured data. Not sure if that's just bad working receiving events through the socket once they occur. – TheMeaningfulEngineer Nov 7 '18 at 16:26
  • Updated the answer to list native journal remote solutions, not involving syslog at all, you might find those interesting! – filbranden Nov 7 '18 at 17:31
  • Thanks. Gave a shot at a pure systemd remote loggin implementation before. I did look promising with the initial PoC, but porting that to a production setting ended up revealing issues that were to complex to cope with at the time. – TheMeaningfulEngineer Nov 8 '18 at 9:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.