Rsyslog `logger` message duplicated

Question

I just had a problem with rsyslog's imjournal module, possibly the issue described here, whereby complications with systemd's journal cause

the journal to return the same data endlessly in a tight loop. This results in massive message duplication inside rsyslog probably resulting in a denial-of-service when the system resouces get exhausted

In any case, it did indeed max the processor out repeating messages that were months old. I did not realize that rsyslog and journald were so tightly coupled by default, so I reconfigured the former to use a more traditional, more efficient live socket for input as per these instructions:

$ModLoad imuxsock
$OmitLocalLogging off

This seems to work in so far as the socket is created and in use. However, I then noticed a strange thing when testing it.

> logger "hello world"

Results in this in /var/log/syslog, which is mentioned only once in rsyslog.conf:

Apr 17 10:35:45 pidora logger: hello world
Apr 17 10:35:45 pidora logger: hello world

The message is repeated, and it would seem that all the other messages are as well. Some of them are exactly the same, and some of them differ in only one aspect:

Apr 17 10:42:26 pidora systemd[1]: Stopping System Time Synchronized.
Apr 17 10:42:26 pidora systemd: Stopping System Time Synchronized.

The [1] is a pid. I believe what's going on is rsyslog is getting the message once from the application and then again from journald.

This is kind of silly. How can I stop it?

Answer 1

I believe what's going on is rsyslog is getting the message once from the application and then again from journald.

Yep. The solution is to include this in /etc/systemd/journald.conf:

ForwardToSyslog=no

Why there wasn't this problem when using imjournal I'm not sure, but there is a hint in man journald.conf:

ForwardToSyslog=

[...] the journal daemon shall be forwarded to a traditional syslog daemon [...] If forwarding to syslog is enabled but no syslog daemon is running, the respective option has no effect

I'm guessing what's actually meant by a "syslog daemon running" is the literal presence of a traditional syslog socket.

Answer 2

I tried this suggestion as well, but I found it had no effect on my system. Rather I found this Comment helpful: syslog duplicate all content

Comment this line .=info;.=notice;.=warn; out, mean #.=info;.=notice;.=warn; . Restart rsyslog.

as Rules from the /etc/rsyslog.conf would be the cause of the phenomenon.

Actually I could prove this with the Tip of the "Hello World" Log Messages

On my System I found these Configurations in /etc/rsyslog.conf

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none                /var/log/messages

# Log services messages
daemon.=error                                           /var/log/kern.log
daemon.*;daemon.!=error                                 /var/log/messages

I added the daemon Facility Rules to register activity from automatization processes.

Now I tried those Test Messages:

$ logger hello my server01 world

and found this Log Message in /var/log/messages

Mar 22 13:35:41 server01 user01: hello my server01 world

But when I tried

$ logger -p daemon.info hello my server01 02

I found these Log Messages:

Mar 22 13:38:15 server01 usre01: hello my server01 02
Mar 22 13:38:15 server01 user01: hello my server01 02

That made me change my Rule in /etc/rsyslog.conf to add a daemon.none Exclusion Rule:

*.info;mail.none;authpriv.none;cron.none;daemon.none     /var/log/messages

and restarted the rsyslog Service. This actually stopped the phenomenon.

Answer 3

In my case, there was below line in rsyslog.conf file

$OmitLocalLogging off

I set it to

$OmitLocalLogging on

Restarted rsyslog service. the duplication stopped. I checked with the logger command like you suggested as well. Thanks very much