syslog-ng service not starting with systemd but command works fine

Question

I have a freshly installed version on CentOS 7 once which I have installed syslog-ng from the EPEL repositories.

~: yum list | grep syslog
syslog-ng.x86_64                        3.5.6-1.el7                    @epel

When I try to start it via systemctl, it fails as follows :

/usr/lib/systemd/system: systemctl start syslog-ng
Job for syslog-ng.service failed. See 'systemctl status syslog-ng.service' and 'journalctl -xn' for details.

When looking into the journals, we can see that their is a dependency on the socket which "starts" fine but that the process returns an error about the arguments being incorrect as shown below :

May 07 17:26:15 superserver.company.corp systemd[1]: Starting Syslog Socket.
May 07 17:26:15 superserver.company.corp systemd[1]: Listening on Syslog Socket.
May 07 17:26:15 superserver.company.corp systemd[1]: Starting System Logger Daemon...
May 07 17:26:15 superserver.company.corp systemd[1]: syslog-ng.service: main process exited, code=exited, status=2/INVALIDARGUMENT
May 07 17:26:15 superserver.company.corp systemd[1]: Failed to start System Logger Daemon.
May 07 17:26:15 superserver.company.corp systemd[1]: Unit syslog-ng.service entered failed state.
May 07 17:26:15 superserver.company.corp systemd[1]: syslog-ng.service holdoff time over, scheduling restart.
May 07 17:26:15 superserver.company.corp systemd[1]: Stopping System Logger Daemon...
May 07 17:26:15 superserver.company.corp systemd[1]: Starting System Logger Daemon...
May 07 17:26:15 superserver.company.corp systemd[1]: syslog-ng.service: main process exited, code=exited, status=2/INVALIDARGUMENT

If we look into the service configuration file, we can confirm the dependency on the socket and the command that is used to start the service.

[Service]
Type=notify
Sockets=syslog.socket
ExecStart=/usr/sbin/syslog-ng -F -p /var/run/syslogd.pid

The problem is that if I run the above-mentionned command, it starts up just fine and it works as expected.

My question is : what is difference between me running the program startup command and systemd starting up the same program ? What can I do to find out what is actually wrong with it ?

---

Edit 1

I enabled the debug output as suggested by Raymond in the answers and the output doesn't teach us much more.

May 08 10:31:29 server.corp systemd[1]: Starting System Logger Daemon...
May 08 10:31:29 server.corp systemd[1]: About to execute: /usr/sbin/syslog-ng -F -p /var/run/syslogd.pid
May 08 10:31:29 server.corp systemd[1]: Forked /usr/sbin/syslog-ng as 3121
May 08 10:31:29 server.corp systemd[1]: syslog-ng.service changed dead -> start
May 08 10:31:29 server.corp systemd[1]: Set up jobs progress timerfd.
May 08 10:31:29 server.corp systemd[1]: Set up idle_pipe watch.
May 08 10:31:29 server.corp systemd[3121]: Executing: /usr/sbin/syslog-ng -F -p /var/run/syslogd.pid
May 08 10:31:29 server.corp systemd[1]: Got notification message for unit syslog-ng.service
May 08 10:31:29 server.corp systemd[1]: syslog-ng.service: Got message
May 08 10:31:29 server.corp systemd[1]: syslog-ng.service: got STATUS=Starting up... (Fri May  8 10:31:29 2015
May 08 10:31:29 server.corp systemd[1]: Got notification message for unit syslog-ng.service
May 08 10:31:29 server.corp systemd[1]: syslog-ng.service: Got message
May 08 10:31:29 server.corp systemd[1]: syslog-ng.service: got STATUS=Starting up... (Fri May  8 10:31:29 2015
May 08 10:31:29 server.corp systemd[1]: Received SIGCHLD from PID 3121 (syslog-ng).
May 08 10:31:29 server.corp systemd[1]: Child 3121 (syslog-ng) died (code=exited, status=2/INVALIDARGUMENT)
May 08 10:31:29 server.corp systemd[1]: Child 3121 belongs to syslog-ng.service
May 08 10:31:29 server.corp systemd[1]: syslog-ng.service: main process exited, code=exited, status=2/INVALIDARGUMENT
May 08 10:31:29 server.corp systemd[1]: syslog-ng.service changed start -> failed
May 08 10:31:29 server.corp systemd[1]: Job syslog-ng.service/start finished, result=failed
May 08 10:31:29 server.corp systemd[1]: Failed to start System Logger Daemon. 

There are a few warnings that are displayed at the start of the syslog-ng processes (nothing that keeps it from starting properly) so I redirected all output to /dev/null but the end result is the same.

Also, as a side note, my entire system does not boot anymore if systemd is unable to syslog. This can be disabled with kernel options to log to kmesg.

Answer 1

We had the same problem on Debian 8.1, but fixed it by changing our syslog-ng local configuration to use unix-dgram instead of unix-socket.

I was clued in by this comment at RedHat Bugzilla:

Note about custom syslog-ng configurations files

People with custom syslog-ng configurations will most likely face upgrade problems due to the unix socket type mismatch between systemd and syslog-ng old configuration file:

• systemd creates /dev/log as unix-dgram
• syslog-ng < 3.2.5 expected /dev/log to be unix-stream (configuration file)

If you use 'unix-stream ("/dev/log")' in one of your log messages sources, you will need to manually change it to 'unix-dgram ("/dev/log")'.

Answer 2

Maybe try adding:

Environment=SYSTEMD_LOG_LEVEL=debug

To your service unit file and then see what dumps out?

I'm also wondering if systemd is attempting to ensure that syslog-ng started successfully by running a

systemctl status syslog-ng

and because there is no matching "status" directive in your unit file it assumes the service didn't startup correctly and kills the process?

Answer 3

I had the exact same problem (Debian 8.4, syslog-ng v3.5.6).

Try to comment or remove the following line in your syslog-ng configuration file:

unix-dgram("/dev/log");

From https://wiki.gentoo.org/wiki/Systemd#syslog-ng_source_for_systemd

Answer 4

fixed by adding this to syslog-ng.service: After=network.target

https://bugzilla.redhat.com/show_bug.cgi?id=1309345

# cat /usr/lib/systemd/system/syslog-ng.service
[Unit]
Description=System Logger Daemon
Documentation=man:syslog-ng(8)
After=network.target

[Service]
Type=notify
Sockets=syslog.socket
ExecStart=/usr/sbin/syslog-ng -F -p /var/run/syslogd.pid
ExecReload=/bin/kill -HUP $MAINPID
StandardOutput=null
Restart=on-failure

[Install]
WantedBy=multi-user.target
Alias=syslog.service

Answer 5

If found that using anything other then ip (0.0.0.0) was reason to crash. So, specifying a known IP address was the problem.

I used syslog-ng 3.5.6.