12

On RHEL7, systemd-journald takes over many of the responsibilites of what was once done by rsyslogd. Whether by bug or conflict between these two daemons, sometimes /dev/log will go missing. As a result, programs relying on the syslog(3) call will not function properly, including, for instance, logger. How can I restore the /dev/log socket?

4 Answers 4

19

Asking and answering my own question because Google was not very helpful on this one.

Normally, with rsyslogd, the imuxsock module will create the /dev/log socket on its own, unlinking the previous entry before creating it. When rsyslogd is stopped (possibly because restart which fails because of faulty configuration), rsyslogd removes /dev/log.

However, the rsyslog supplied with RHEL7 is expected to be used in conjunction with systemd, and the imuxsock module will actually open and remove /run/systemd/journal/syslog socket. Meanwhile, the /dev/log device is created by the system service-file systemd-journald.socket which triggers journald.

Apparently, whether or not $imjournal module is used, the following works.

In sum, if /dev/log disappears:

  1. restart systemd-journald.socket:

    systemctl restart systemd-journald.socket
    
  2. then restart rsyslogd

    systemctl start rsyslogd
    

UPDATE: I believe systemctl restart rsyslogd might re-delete the socket if rsyslogd is already running.

1
  • 3
    Thanks so much for this! I just lost several hours tracking down why logging wasn't working for a service. I finally tracked it down to the missing /dev/log, which lead me to your solution. Dec 9, 2016 at 3:23
4

The systemctl restart systemd-journald.socket && systemctl restart rsyslog solution did not work for me on Ubuntu 16.04.

Instead, I had to recreate /dev/log as a symlink to /run/systemd/journal/dev-log:

ln -s /run/systemd/journal/dev-log /dev/log
4
  • Yes, manually linking should work as well. But it's a bit unwieldy to remember. Question: On Ubuntu does systemd-journald.socket exist as a service? Q2: maybe the restart rsyslogd was the problem? Maybe it should simply be start rsyslogd?
    – Otheus
    Mar 9, 2017 at 14:13
  • Q1: yes, it exists. Q2: there's no restart command anymore, there's service and /etc/init.d/rsyslog.
    – 11181
    Mar 9, 2017 at 21:46
  • By restart rsyslogd I thought it was clear I meant systemctl restart rsyslogd. Does Ubuntu still use init scripts for rsyslog?
    – Otheus
    Mar 10, 2017 at 8:52
  • @Otheus /etc/init.d/rsyslog stop followed by /etc/init.d/rsyslog start did not help. Neither did systemctl stop syslog.socket rsyslog.service && systemctl start syslog.socket rsyslog.service On my system there's both /lib/systemd/system/rsyslog.service and /etc/init.d/rsyslog. Anyway, I'd rather not spend more time on this problem.
    – 11181
    Mar 11, 2017 at 19:03
1

For me this ended up being a problem with how the imuxsock module used in rsyslog was working with systemd.

In the imuxsock documentation they walk through how the module is supposed to work for systemd. Step 1 was where I was seeing issues:

Step 1: Select name of system socket

  1. If the user has not explicitly chosen to set SysSock.Use="off" then the default listener socket (aka, “system log socket” or simply “system socket”) name is set to /dev/log. Otherwise, if the user has explicitly set SysSock.Use="off", then rsyslog will not listen on /dev/log OR any socket defined by the SysSock.Name parameter and the rest of this section does not apply.

  2. If the user has specified sysSock.Name="/path/to/custom/socket" (and not explicitly set SysSock.Use="off"), then the default listener socket name is overwritten with /path/to/custom/socket.

  3. Otherwise, if rsyslog is running under systemd AND /run/systemd/journal/syslog exists, (AND the user has not explicitly set SysSock.Use="off") then the default listener socket name is overwritten with /run/systemd/journal/syslog.

The system should have fallen through to Step 3 and changed the default path to be "/run/systemd/journal/syslog", but instead it stayed "/var/log". This meant that the imuxsock module would try (and succeed sometimes) to create a socket at /dev/log where there should instead be symbolic link created by the systemd-journald-dev-log.socket. In the case that it would fail to create the real socket, the symbolic link would still be removed.

That documentation was the outcome of this issue reported on the rsyslog github. If you want to skip the discussion and jump straight to the changes see PR#1 and PR#2 respectively.

My solution was to just configure the imuxsock module to use the systemd path in my /etc/rsyslog.conf:

module(load="imuxsock"
    SysSock.Name="/run/systemd/journal/syslog")

This seems to have fixed my issue and sounds like a good solution here since it would explain why the symbolic link might disappear again after you would manually create it.

If you look on your system and "/run/systemd/journal/syslog" is not present look at the "syslog.socket" to see if it is starting successfully as that is what is responsible for creating the socket.

systemctl status syslog.socket

It could be that your version of rsyslog.service doesn't define syslog.service as an alias which is needed as the syslog.socket tries to active that service. It is also possible that multiple logging services try to alias syslog.service in which case last to be enabled wins.

1
  • Thanks for that thorough answer. It's mind-boggling how convoluted this all is. I blame systemd. :-D
    – Otheus
    Apr 7 at 11:25
0

I am not sure how the nuts and bolts of systemd and rsyslog work, but I tinkered on it for a while, and found why /dev/log disappears and how to restore it.

As mentioned in previous answers, Rsyslog will generate /dev/log socket by itself when the service starts if the imuxsock module is set in /etc/rsyslog.conf, and remove the socket once the service stops. According to the imuxsock documentation, rsyslog will only remove the socket if it was NOT passed in via systemd. In this case, /dev/log socket was passed in with rsyslog and not systemd, thus, it removes the socket if the service stops.

By default (I am using Fedora 36), rsyslog uses imuxsock module but with the "SysSock.Use=off" parameter. With this, rsyslog will not use the system log socket (/dev/log) as its socket, and thus, not generate /dev/log. Using this configuration (module(load="imuxsock" SysSock.use="off")) in the /etc/rsyslog.conf will result in rsyslog not logging anything.

However, there is also another module loaded by default, which is the imjournal module. This gains rsyslog access to the systemd journal. From what I understood, rsyslog will not be the one generating /dev/log socket, rather systemd-journal-dev-log.socket will. This /dev/log socket will be symlinked to /run/systemd/journal/dev-log. Even if module(load="imuxsock" SysSock.use="off") is used, rsyslog will still log activities since it pulls data from systemd journal. Now that /dev/log -> /run/systemd/journal/dev-log is created, rsyslog will not remove this since the socket was passed in via systemd.

From what I have tried, /dev/log could disappear because "SysSock.use=off" parameter is later added or uncommented in /etc/rsyslog.conf (even if imjournal module was loaded). This tells rsyslog to not generate /dev/log.

You could say that adding the imjournal module again and restarting the service would solve this problem; however, the confusing part is that this does not work and no /dev/log -> /run/systemd/journal/dev-log can be found. This is because the systemd-journald-dev-log.socket service was not aware that /dev/log socket is missing. Thus, restarting this service would trigger it that the socket is missing, then restarting the systemd-journald.socket afterwards will solve this problem. Although I am not sure that restarting the systemd-journald-dev-log.socket service is okay, this brings the /dev/log socket back.

In summary, do the following to get the /dev/log socket back:

  1. Make sure you have these in /etc/rsyslog.conf:
    • module(load="imuxsock" SysSock.use="off")
    • module(load="imjournal" StateFile="imjournal.state")
  2. Restart rsyslog
    # systemctl restart rsyslog
    
  3. Restart systemd-journald-dev-log.socket (you may see 'Job failed. See "journalctl -xe" for details.' warning, but I think it is okay since it stops and starts immediately)
    # systemctl restart systemd-journald-dev-log.socket
    
  4. Restart systemd-journald.socket
    # systemctl restart systemd-journald.socket
    

Note 1: From what I understand, this works only with RHEL system. Debian or Ubuntu-based systems may not have this problem, or different ways to solve it

Note 2: I am sorry for the long and confusing answer.  This is my first time here and I am still a newbie :D

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.