293

I have created a simple systemd service file for a custom application. The application works well when I run it manually, but my CPU gets maxed out when I run it with systemd.

I'm trying do track down where my problem is, but I don't know where to find the output (or how to configure systemd to put the output somewhere).

Here is my service file:

[Unit]
Description=Syncs files with a server when they change
Wants=network.target
After=network.target

[Service]
ExecStart=/usr/local/bin/filesync-client --port 2500
WorkingDirectory=/usr/local/lib/node_modules/filesync-client
Restart=always

[Install]
WantedBy=multi-user.target

Throughout the application, I output to stdout and stderr.

How can I read the output of my daemon?

Edit:

I found man systemd.exec, which mentioned the StandardOutput= option, but I'm not sure how to use it. From the man page:

StandardOutput=

Controls where file descriptor 1 (STDOUT) of the executed processes is connected to. Takes one of inherit, null, tty, syslog, kmsg, kmsg+console, syslog+console or socket.

If set to inherit the file descriptor of standard input is duplicated for standard output. If set to null standard output will be connected to /dev/null, i.e. everything written to it will be lost. If set to tty standard output will be connected to a tty (as configured via TTYPath=, see below). If the TTY is used for output only the executed process will not become the controlling process of the terminal, and will not fail or wait for other processes to release the terminal. syslog connects standard output to the syslog(3) system logger. kmsg connects it with the kernel log buffer which is accessible via dmesg(1). syslog+console and kmsg+console work similarly but copy the output to the system console as well. socket connects standard output to a socket from socket activation, semantics are similar to the respective option of StandardInput=. This setting defaults to inherit.

Does this mean that these are my only options? I would like, for example, to put output in /dev/shm or something. I suppose I could use a Unix domain socket and write a simple listener, but this seems a little unnecessary.

I just need this for debugging, and I'll probably end up removing most of the logs and change the output to syslog.

7
  • Have you tried checking /var/log/syslog for output? Most systems will log stuff into /var/log/ so I'd start by checking there. You can use grep to search for text if you know the output: grep "my output" /var/log should do the trick.
    – sbtkd85
    Sep 9, 2011 at 19:10
  • @sbtkd85 - Well, I don't have /var/log/syslog, but /var/log/messages does the trick. The problem is, according to the logs, my daemon crashes on start, yet I can tell that it is still running because it has an HTTP server, and I can query it. It seems the rest of the logs are getting lost...
    – beatgammit
    Sep 10, 2011 at 3:50
  • Why not try setting StandardOutput=tty so you can see what is happening when you launch your daemon. It should output the terminal (you may have to use ttyS0 or similar to get the output on your screen).
    – sbtkd85
    Sep 12, 2011 at 13:46
  • 5
    Shouldn't standard IO redirection operators work in this context. Something like ExecStart=/usr/local/bin/filesync-client --port 2500 2>/tmp/filesync.log
    – Deepak Mittal
    Apr 10, 2012 at 18:30
  • What is actually abusing your CPU? Is it systemd, your service or system (e.g. by spawning new copies of the service because systemd went crazy)?
    – peterph
    Nov 13, 2012 at 16:03

2 Answers 2

Reset to default
294

Update

As mikemaccana notes, the systemd journal is now the standard logging device for most distros. To view the stdout and stderr of a systemd unit use the journalctl command.

sudo journalctl -u [unit]

Original Answer

By default stdout and stderr of a systemd unit are sent to syslog.

If you're using the full systemd, this will be accesible via journalctl. On Fedora, it should be /var/log/messages but syslog will put it where your rules say.

Due to the date of the post, and assuming most people that are exposed to systemd are via fedora, you were probably hit by the bug described here: https://bugzilla.redhat.com/show_bug.cgi?id=754938 It has a good explanation of how it all works too =) (This was a bug in selinux-policy that caused error messages to not be logged, and was fixed in selinux-policy-3.10.0-58.fc16)

10
  • 5
    Note that using the standard logging mechanism like this will not create persistent logs by default. To do that, you'll need to create /var/log/journal, and then run sudo systemctl restart systemd-journald
    – mlissner
    May 8, 2015 at 2:21
  • 1
    what syslog facility and priority?
    – jrwren
    Jul 7, 2015 at 16:31
  • 2
    This worked for me: StandardOutput=syslog+console and StandardError=syslog+console after that all output from my unit appeared in journalctl. The default setting was wrong apparently. (Such as DefaultStandardOutput in /etc/systemd/system.conf)
    – gregn3
    Jul 22, 2015 at 12:23
  • 2
    -f was helpful for me. Followed the log as changes occur (use case was following minecraft server which was running as a daemon)
    – blaughw
    Nov 21, 2016 at 4:55
  • 3
    This is driving me crazy... On a standard Debian stretch journalctl won't show me any of the standard output whatsoever. I even use /usr/bin/stdbuf -oL <cmd> and an explicit StandardOutput=journal. Still nothing.
    – jlh
    Dec 18, 2018 at 9:14
123

Shorter, simpler, non-legacy answer:

sudo journalctl -u [unitfile]

Where [unitfile] is the systemd .service name. Eg, to see messages from myapp.service, 

sudo journalctl --unit=myapp

To follow logs in real time: 

sudo journalctl -f -u myapp
8
  • 5
    Note that you might have to sudo if you get a No journal files found error.
    – bigjosh
    Nov 17, 2015 at 15:51
  • 9
    syslog is not legacy...
    – Miles Rout
    Jun 26, 2016 at 5:18
  • 4
    It is in current Linux distros. You might really like syslog but that doesn't change what they ship with.
    – mikemaccana
    Jun 26, 2016 at 11:13
  • 12
    @JECompton if all logging on current Linux distros uses journald, and syslog is not necessary and only used for compatibility, then it logically follows that syslog is legacy.
    – mikemaccana
    Mar 13, 2017 at 11:27
  • 5
    @mikemaccana Nonsene. Debian/Ubuntu/RedHat != "all logging on current Linux". syslogd is very much alive and well and will continue to be relevant for a long long time. Just one example: BusyBox runs on hundreds of millions, if not billions of embedded devices the world over - far outstripping any desktop distro in popularity. Its system logging facility? Yeah. There are also many desktop distros which have managed to evade the SystemD malware infection, such as Devuan, Alpine Linux, Artix Linux, Gentoo, Slackware, etc - many of which will be using syslogd.
    – Ola Tuvesson
    Sep 13, 2019 at 9:11

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .