On RHEL7, systemd-journald
takes over many of the responsibilites of what was once done by rsyslogd
. Whether by bug or conflict between these two daemons, sometimes /dev/log
will go missing. As a result, programs relying on the syslog(3)
call will not function properly, including, for instance, logger
. How can I restore the /dev/log
socket?
Asking and answering my own question because Google was not very helpful on this one.
Normally, with rsyslogd
, the imuxsock
module will create the /dev/log
socket on its own, unlinking the previous entry before creating it. When rsyslogd
is stopped (possibly because restart which fails because of faulty configuration), rsyslogd removes /dev/log
.
However, the rsyslog supplied with RHEL7
is expected to be used in conjunction with systemd
, and the imuxsock
module will actually open and remove /run/systemd/journal/syslog
socket. Meanwhile, the /dev/log
device is created by the system service-file systemd-journald.socket
which triggers journald
.
Apparently, whether or not $imjournal
module is used, the following works.
In sum, if /dev/log
disappears:
restart systemd-journald.socket:
systemctl restart systemd-journald.socket
then restart rsyslogd
systemctl start rsyslogd
UPDATE: I believe restart rsyslogd
might re-delete the socket if rsyslogd
is already running.
-
2Thanks so much for this! I just lost several hours tracking down why logging wasn't working for a service. I finally tracked it down to the missing /dev/log, which lead me to your solution. – Chad Huneycutt Dec 9 '16 at 3:23
The systemctl restart systemd-journald.socket && systemctl restart rsyslog
solution did not work for me on Ubuntu 16.04.
Instead, I had to recreate /dev/log
as a symlink to /run/systemd/journal/dev-log
:
ln -s /run/systemd/journal/dev-log /dev/log
-
Yes, manually linking should work as well. But it's a bit unwieldy to remember. Question: On Ubuntu does
systemd-journald.socket
exist as a service? Q2: maybe therestart rsyslogd
was the problem? Maybe it should simply bestart rsyslogd
? – Otheus Mar 9 '17 at 14:13 -
Q1: yes, it exists. Q2: there's no
restart
command anymore, there'sservice
and/etc/init.d/rsyslog
. – 11181 Mar 9 '17 at 21:46 -
By
restart rsyslogd
I thought it was clear I meantsystemctl restart rsyslogd
. Does Ubuntu still use init scripts for rsyslog? – Otheus Mar 10 '17 at 8:52 -
@Otheus
/etc/init.d/rsyslog stop
followed by/etc/init.d/rsyslog start
did not help. Neither didsystemctl stop syslog.socket rsyslog.service && systemctl start syslog.socket rsyslog.service
On my system there's both/lib/systemd/system/rsyslog.service
and/etc/init.d/rsyslog
. Anyway, I'd rather not spend more time on this problem. – 11181 Mar 11 '17 at 19:03
For me this ended up being a problem with how the imuxsock module used in rsyslog was working with systemd.
In the imuxsock documentation they walk through how the module is supposed to work for systemd. Step 1 was where I was seeing issues:
Step 1: Select name of system socket
If the user has not explicitly chosen to set SysSock.Use="off" then the default listener socket (aka, “system log socket” or simply “system socket”) name is set to /dev/log. Otherwise, if the user has explicitly set SysSock.Use="off", then rsyslog will not listen on /dev/log OR any socket defined by the SysSock.Name parameter and the rest of this section does not apply.
If the user has specified sysSock.Name="/path/to/custom/socket" (and not explicitly set SysSock.Use="off"), then the default listener socket name is overwritten with /path/to/custom/socket.
Otherwise, if rsyslog is running under systemd AND /run/systemd/journal/syslog exists, (AND the user has not explicitly set SysSock.Use="off") then the default listener socket name is overwritten with /run/systemd/journal/syslog.
The system should have falling into Step 3 and changing the default path to be "/run/systemd/journal/syslog" but instead it was remaining "/var/log". This meant that the imuxsock module would try (and succeed sometimes) to create a socket at /dev/log where there should instead be symbolic link created by the systemd-journald-dev-log.socket. In the case that it would fail to create the real socket, the symbolic link would still be removed.
That documentation was the outcome of this issue reported on the rsyslog github. If you want to skip the discussion and jump straight to the changes see PR#1 and PR#2 respectively.
My solution was to just configure the imuxsock module to use the systemd path in my /etc/rsyslog.conf:
module(load="imuxsock"
SysSock.Name="/run/systemd/journal/syslog")
This seems to have fixed my issue and sounds like a good solution here since it would explain why the symbolic link might disappear again after you would manually create it.
If you look on your system and "/run/systemd/journal/syslog" is not present look at the "syslog.socket" to see if it is starting successfully as that is what is responsible for creating the socket.
systemctl status syslog.socket
It could be that your version of rsyslog.service doesn't define syslog.service as an alias which is needed as the syslog.socket tries to active that service. It is also possible that multiple logging services try to alias syslog.service in which case last to be enabled wins.