How do I get the standard output/error of a service, controlled by systemd on EL7, to go somewhere other than /var/log/messages?

Question

My use cases are what Fabrice A. Marie's Logs duplicates with systemd article (published by Kibin Labs) places into its "ugly" category, and its brief advice to add in an entire ELK stack (it's the "only recourse", it says) for "ugly" is overkill for my needs.

Fabrice A. Marie's categorization is that "good" applications can have standard output redirected to /dev/null without resultant loss of information, only write very little to standard error and hence to /var/log/messages (via the systemd journal), and send the bulk of their logs through per-application logging mechanisms (unmanaged by systemd or rsyslog) to private application-maintained log files. This is, if anything, the reverse of what I want, and not "good" by my criteria.

I have several applications (one of which, picked for the sake of example here, is phantomjs backend prerendering, but this is just one of my use cases) that are launched as services under systemd. My applications send their log to standard output and standard error (this being the normal default behaviour of console.log() in node.js). It seems wrong that to make these qualify as "good" they should be rewritten to do their own private logging, instead of using systemd for that (as they are now). In any case, I'm not in a position to re-code these applications myself.

I appear, going by what Fabrice A. Marie says and by what else I have read, to have the stark choice of all of that standard output/standard error information going either to /dev/null or to /var/log/messages. Is there no happy median? Can I not set up systemd (and whatever else) so that my applications' logs, being written to standard output/error, are handled by systemd in the normal way, without an entire ELK stack in the middle, and yet end up in another place than /var/log/messages?

I'm running both CentOS 7 and RHEL 7, and the service unit for my example service is currently this:

[Unit]
Description=prerender
After=network.target
[Service]
ExecStart=/usr/local/bin/node /var/www/prerender/node/server.js
Restart=always
User=prerender
Group=adm
Environment=PATH=/usr/bin:/usr/local/bin PORT=8900
WorkingDirectory=/var/www/prerender/node
[Install]
WantedBy=multi-user.target

Answer 1

That article looks like nonsense, putting the blame on other software for problems caused by systemd's hostile takeover of logging functions. It even mentions the solution (syslog) but only as a means to suggesting massive overkill like logstash.

You don't need logstash to log a handful of daemons on a single host.

From the article:

And by default when it does this, stdout is redirected to the journal. The journal in turn is sent to syslog by default, which on RedHat 7 compatible distros will end up in /var/log/messages.

It then completely ignores the obvious and sensible option of configuring rsyslogd and implies that logstash and elasticsearch are the only options if you want to do something that journald can't do.

Modern syslog daemons, such as rsyslog and syslog-ng can be configured to filter syslog messages by all sorts of criteria, including daemon name and regex pattern match. The default may be /var/log/messages, but it is trivially easy to change the default.

So what you need are some rsyslog rules for filtering syslog messages from your service.

---

Take whatever systemd advocates say about logging with a huge grain of salt - they like to pretend that journald was necessary because logging before that was primitive and inflexible, that they were solving a problem nobody had even thought about before let alone solved several times over.

Also, BTW, the correct way for a program to do logging is neither stdout nor stderr, it's by using the syslog functions that are available in most languages.