How should I send systemd logs to a dedicated logging server

Question

In the past, I have used syslog-ng or rsyslog to send system logs to a dedicated logging server somewhere else on the network. With systemd however, things like the sudo commands that previously were stored in /var/log/secure or /var/log/auth.log (depending on your distro) are now stored in the journal.

I am aware that /etc/systemd/journald.conf has a ForwardToSyslog=yes option that would allow me to return to my old ways, but is this seems like an inelegant step backwards. Does systemd have a built-in means of sending logs to a central logging server like Logrhythm, ELK or similar, or setting ForwardToSyslog=yes the proper way to do this?

Answer 1

It appears that systemd does not have a built-in means of forwarding messages to a syslog server. Red Hat's official recommendation is to use the imjournal module to allow rsyslog to read the journald logs and forward these logs to a central logging server by setting the following in /etc/rsyslog.conf:

module(load="imjournal"
    PersistStateInterval="number_of_messages"
    StateFile="path"
    ratelimit.interval="seconds"
    ratelimit.burst="burst_number"
    IgnorePreviousMessages="off/on")

They provide usage details for these options , noting

• With number_of_messages, you can specify how often the journal data must be saved. This will happen each time the specified number of messages is reached.

• Replace path with a path to the state file. This file tracks the journal entry that was the last one processed.

• With seconds, you set the length of the rate limit interval. The number of messages processed during this interval can not exceed the value specified in burst_number. The default setting is 20,000 messages per 600 seconds. Rsyslog discards messages that come after the maximum burst within the time frame specified.

• With IgnorePreviousMessages you can ignore messages that are currently in Journal and import only new messages, which is used when there is no state file specified. The default setting is off. Please note that if this setting is off and there is no state file, all messages in the Journal are processed, even if they were already processed in a previous rsyslog session.

Answer 2

I did not read thoroughly about imjournald before but After reading James Shewey's answer, I think it is really easy.

Using rulesets, you can achieve what ever you want.

module(load="imjournal"
    PersistStateInterval="number_of_messages"
    StateFile="path"
    ratelimit.interval="seconds"
    ratelimit.burst="burst_number"
    IgnorePreviousMessages="off/on"
    ruleset="journald"  #bind this action to the ruleset "journald"
    )

#declare ruleset "journald"

ruleset(
 name="journald"              
 queue.type="FixedArray"       
 queue.spoolDirectory="/var/run/rsyslog/queues" 
 queue.filename="stats_ruleset"
 queue.lowwatermark="20000"     
 queue.maxdiskspace="100m"        
 queue.size="5000000"             
 queue.dequeuebatchsize="1000"  
 queue.saveonshutdown="on"      
) {

# add here actions you want to perform

action(
  name="impstats_to_es"
  type="omelasticsearch"
  server="127.0.0.1"
  serverport="9200"
  template="stats"
  searchIndex="impstats"
  searchType="impstats"
  bulkmode="on"
  action.resumeretrycount="-1"
)

}