6

I have a freshly installed version on CentOS 7 once which I have installed syslog-ng from the EPEL repositories.

~: yum list | grep syslog
syslog-ng.x86_64                        3.5.6-1.el7                    @epel

When I try to start it via systemctl, it fails as follows :

/usr/lib/systemd/system: systemctl start syslog-ng
Job for syslog-ng.service failed. See 'systemctl status syslog-ng.service' and 'journalctl -xn' for details.

When looking into the journals, we can see that their is a dependency on the socket which "starts" fine but that the process returns an error about the arguments being incorrect as shown below :

May 07 17:26:15 superserver.company.corp systemd[1]: Starting Syslog Socket.
May 07 17:26:15 superserver.company.corp systemd[1]: Listening on Syslog Socket.
May 07 17:26:15 superserver.company.corp systemd[1]: Starting System Logger Daemon...
May 07 17:26:15 superserver.company.corp systemd[1]: syslog-ng.service: main process exited, code=exited, status=2/INVALIDARGUMENT
May 07 17:26:15 superserver.company.corp systemd[1]: Failed to start System Logger Daemon.
May 07 17:26:15 superserver.company.corp systemd[1]: Unit syslog-ng.service entered failed state.
May 07 17:26:15 superserver.company.corp systemd[1]: syslog-ng.service holdoff time over, scheduling restart.
May 07 17:26:15 superserver.company.corp systemd[1]: Stopping System Logger Daemon...
May 07 17:26:15 superserver.company.corp systemd[1]: Starting System Logger Daemon...
May 07 17:26:15 superserver.company.corp systemd[1]: syslog-ng.service: main process exited, code=exited, status=2/INVALIDARGUMENT

If we look into the service configuration file, we can confirm the dependency on the socket and the command that is used to start the service.

[Service]
Type=notify
Sockets=syslog.socket
ExecStart=/usr/sbin/syslog-ng -F -p /var/run/syslogd.pid

The problem is that if I run the above-mentionned command, it starts up just fine and it works as expected.

My question is : what is difference between me running the program startup command and systemd starting up the same program ? What can I do to find out what is actually wrong with it ?


Edit 1

I enabled the debug output as suggested by Raymond in the answers and the output doesn't teach us much more.

May 08 10:31:29 server.corp systemd[1]: Starting System Logger Daemon...
May 08 10:31:29 server.corp systemd[1]: About to execute: /usr/sbin/syslog-ng -F -p /var/run/syslogd.pid
May 08 10:31:29 server.corp systemd[1]: Forked /usr/sbin/syslog-ng as 3121
May 08 10:31:29 server.corp systemd[1]: syslog-ng.service changed dead -> start
May 08 10:31:29 server.corp systemd[1]: Set up jobs progress timerfd.
May 08 10:31:29 server.corp systemd[1]: Set up idle_pipe watch.
May 08 10:31:29 server.corp systemd[3121]: Executing: /usr/sbin/syslog-ng -F -p /var/run/syslogd.pid
May 08 10:31:29 server.corp systemd[1]: Got notification message for unit syslog-ng.service
May 08 10:31:29 server.corp systemd[1]: syslog-ng.service: Got message
May 08 10:31:29 server.corp systemd[1]: syslog-ng.service: got STATUS=Starting up... (Fri May  8 10:31:29 2015
May 08 10:31:29 server.corp systemd[1]: Got notification message for unit syslog-ng.service
May 08 10:31:29 server.corp systemd[1]: syslog-ng.service: Got message
May 08 10:31:29 server.corp systemd[1]: syslog-ng.service: got STATUS=Starting up... (Fri May  8 10:31:29 2015
May 08 10:31:29 server.corp systemd[1]: Received SIGCHLD from PID 3121 (syslog-ng).
May 08 10:31:29 server.corp systemd[1]: Child 3121 (syslog-ng) died (code=exited, status=2/INVALIDARGUMENT)
May 08 10:31:29 server.corp systemd[1]: Child 3121 belongs to syslog-ng.service
May 08 10:31:29 server.corp systemd[1]: syslog-ng.service: main process exited, code=exited, status=2/INVALIDARGUMENT
May 08 10:31:29 server.corp systemd[1]: syslog-ng.service changed start -> failed
May 08 10:31:29 server.corp systemd[1]: Job syslog-ng.service/start finished, result=failed
May 08 10:31:29 server.corp systemd[1]: Failed to start System Logger Daemon. 

There are a few warnings that are displayed at the start of the syslog-ng processes (nothing that keeps it from starting properly) so I redirected all output to /dev/null but the end result is the same.

Also, as a side note, my entire system does not boot anymore if systemd is unable to syslog. This can be disabled with kernel options to log to kmesg.

  • did you manage to solve this issue? Please post the solution on centos7 if it was resolved – satch_boogie Aug 21 '16 at 4:57
6

We had the same problem on Debian 8.1, but fixed it by changing our syslog-ng local configuration to use unix-dgram instead of unix-socket.

I was clued in by this comment at RedHat Bugzilla:

Note about custom syslog-ng configurations files

People with custom syslog-ng configurations will most likely face upgrade problems due to the unix socket type mismatch between systemd and syslog-ng old configuration file:

  • systemd creates /dev/log as unix-dgram
  • syslog-ng < 3.2.5 expected /dev/log to be unix-stream (configuration file)

If you use 'unix-stream ("/dev/log")' in one of your log messages sources, you will need to manually change it to 'unix-dgram ("/dev/log")'.

1

I had the exact same problem (Debian 8.4, syslog-ng v3.5.6).

Try to comment or remove the following line in your syslog-ng configuration file:

unix-dgram("/dev/log");

From https://wiki.gentoo.org/wiki/Systemd#syslog-ng_source_for_systemd

0

Maybe try adding:

Environment=SYSTEMD_LOG_LEVEL=debug

To your service unit file and then see what dumps out?

I'm also wondering if systemd is attempting to ensure that syslog-ng started successfully by running a

systemctl status syslog-ng

and because there is no matching "status" directive in your unit file it assumes the service didn't startup correctly and kills the process?

  • Thanks for your reply. I added the debug line configuration and it added nothing. Now my system hangs at boot with no debug at all :-/ – Antoine Benkemoun May 8 '15 at 8:08
  • Ah very interesting, systemd doesn't boot if it can't log into syslog-ng... – Antoine Benkemoun May 8 '15 at 8:26
0

fixed by adding this to syslog-ng.service: After=network.target

https://bugzilla.redhat.com/show_bug.cgi?id=1309345

# cat /usr/lib/systemd/system/syslog-ng.service
[Unit]
Description=System Logger Daemon
Documentation=man:syslog-ng(8)
After=network.target

[Service]
Type=notify
Sockets=syslog.socket
ExecStart=/usr/sbin/syslog-ng -F -p /var/run/syslogd.pid
ExecReload=/bin/kill -HUP $MAINPID
StandardOutput=null
Restart=on-failure

[Install]
WantedBy=multi-user.target
Alias=syslog.service
-1

If found that using anything other then ip (0.0.0.0) was reason to crash. So, specifying a known IP address was the problem.

I used syslog-ng 3.5.6.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.