168

I have created a simple systemd service file for a custom application. The application works well when I run it manually, but my CPU gets maxed out when I run it with systemd.

I'm trying do track down where my problem is, but I don't know where to find the output (or how to configure systemd to put the output somewhere).

Here is my service file:

[Unit]
Description=Syncs files with a server when they change
Wants=network.target
After=network.target

[Service]
ExecStart=/usr/local/bin/filesync-client --port 2500
WorkingDirectory=/usr/local/lib/node_modules/filesync-client
Restart=always

[Install]
WantedBy=multi-user.target

Throughout the application, I output to stdout and stderr.

How can I read the output of my daemon?

Edit:

I found man systemd.exec, which mentioned the StandardOutput= option, but I'm not sure how to use it. From the man page:

StandardOutput=

Controls where file descriptor 1 (STDOUT) of the executed processes is connected to. Takes one of inherit, null, tty, syslog, kmsg, kmsg+console, syslog+console or socket.

If set to inherit the file descriptor of standard input is duplicated for standard output. If set to null standard output will be connected to /dev/null, i.e. everything written to it will be lost. If set to tty standard output will be connected to a tty (as configured via TTYPath=, see below). If the TTY is used for output only the executed process will not become the controlling process of the terminal, and will not fail or wait for other processes to release the terminal. syslog connects standard output to the syslog(3) system logger. kmsg connects it with the kernel log buffer which is accessible via dmesg(1). syslog+console and kmsg+console work similarly but copy the output to the system console as well. socket connects standard output to a socket from socket activation, semantics are similar to the respective option of StandardInput=. This setting defaults to inherit.

Does this mean that these are my only options? I would like, for example, to put output in /dev/shm or something. I suppose I could use a Unix domain socket and write a simple listener, but this seems a little unnecessary.

I just need this for debugging, and I'll probably end up removing most of the logs and change the output to syslog.

  • Have you tried checking /var/log/syslog for output? Most systems will log stuff into /var/log/ so I'd start by checking there. You can use grep to search for text if you know the output: grep "my output" /var/log should do the trick. – sbtkd85 Sep 9 '11 at 19:10
  • @sbtkd85 - Well, I don't have /var/log/syslog, but /var/log/messages does the trick. The problem is, according to the logs, my daemon crashes on start, yet I can tell that it is still running because it has an HTTP server, and I can query it. It seems the rest of the logs are getting lost... – beatgammit Sep 10 '11 at 3:50
  • Why not try setting StandardOutput=tty so you can see what is happening when you launch your daemon. It should output the terminal (you may have to use ttyS0 or similar to get the output on your screen). – sbtkd85 Sep 12 '11 at 13:46
  • 3
    Shouldn't standard IO redirection operators work in this context. Something like ExecStart=/usr/local/bin/filesync-client --port 2500 2>/tmp/filesync.log – Deepak Mittal Apr 10 '12 at 18:30
  • What is actually abusing your CPU? Is it systemd, your service or system (e.g. by spawning new copies of the service because systemd went crazy)? – peterph Nov 13 '12 at 16:03
177

Update

As mikemaccana notes, the systemd journal is now the standard logging device for most distros. To view the stdout and stderr of a systemd unit use the journalctl command.

sudo journalctl -u [unit]

Original Answer

By default stdout and stderr of a systemd unit are sent to syslog.

If you're using the full systemd, this will be accesible via journalctl. On Fedora, it should be /var/log/messages but syslog will put it where your rules say.

Due to the date of the post, and assuming most people that are exposed to systemd are via fedora, you were probably hit by the bug described here: https://bugzilla.redhat.com/show_bug.cgi?id=754938 It has a good explanation of how it all works too =) (This was a bug in selinux-policy that caused error messages to not be logged, and was fixed in selinux-policy-3.10.0-58.fc16)

  • 5
    Note that using the standard logging mechanism like this will not create persistent logs by default. To do that, you'll need to create /var/log/journal, and then run sudo systemctl restart systemd-journald – mlissner May 8 '15 at 2:21
  • 1
    what syslog facility and priority? – jrwren Jul 7 '15 at 16:31
  • 2
    This worked for me: StandardOutput=syslog+console and StandardError=syslog+console after that all output from my unit appeared in journalctl. The default setting was wrong apparently. (Such as DefaultStandardOutput in /etc/systemd/system.conf) – gregn3 Jul 22 '15 at 12:23
  • 2
    -f was helpful for me. Followed the log as changes occur (use case was following minecraft server which was running as a daemon) – blaughw Nov 21 '16 at 4:55
  • 1
    This is driving me crazy... On a standard Debian stretch journalctl won't show me any of the standard output whatsoever. I even use /usr/bin/stdbuf -oL <cmd> and an explicit StandardOutput=journal. Still nothing. – jlh Dec 18 '18 at 9:14
75

Shorter, simpler, non-legacy answer:

sudo journalctl -u [unitfile]

Where [unitfile] is the systemd .service name. Eg, to see messages from myapp.service,

sudo journalctl --unit=myapp

To follow logs in real time:

sudo journalctl -f -u myapp
  • 4
    Note that you might have to sudo if you get a No journal files found error. – bigjosh Nov 17 '15 at 15:51
  • 4
    syslog is not legacy... – Miles Rout Jun 26 '16 at 5:18
  • 2
    It is in current Linux distros. You might really like syslog but that doesn't change what they ship with. – mikemaccana Jun 26 '16 at 11:13
  • So Ubuntu 16.04 LTS isn't a current distro in 2017? Maybe you're conflating the idea of sysvinit with syslog. – labyrinth Mar 9 '17 at 23:08
  • 6
    @JECompton if all logging on current Linux distros uses journald, and syslog is not necessary and only used for compatibility, then it logically follows that syslog is legacy. – mikemaccana Mar 13 '17 at 11:27

protected by Stephen Kitt Jul 18 '18 at 18:17

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.