Combine svlogd logs from different runit services

Question

I want to run a Node.js app with multiple Node processes running the exact same code. I'm using runit to start and manage the Node processes, because it has good Chef support. To illustrate the set up, imagine I'm running 2 runit service directories:

/etc/sv/test-1/run

#!/bin/sh
echo running test 1
sleep 9999

/etc/sv/test-2/run

#!/bin/sh
echo running test 2
sleep 9999

(Where echo and sleep are test placeholders for running the actual Node app.)

This works great... except by default runit's svlogd program wants to put each service's logs in separate directories. The problem is, I don't want to have to look in multiple places to see all the log entries for one application.

My first attempt at getting svlogd to write to the same log file looks like this:

/etc/sv/test-1/log/run

#!/bin/sh
exec svlogd -tt /var/log/test

/etc/sv/test-2/log/run

#!/bin/sh
exec svlogd -tt /var/log/test

Only this doesn't work. Whichever log service starts first, runs fine. The log service that starts second does not run because it is locked out by the other log service. To illustrate:

$ sudo /etc/service/test-2/log/run
svlogd: warning: unable to lock directory: /var/log/test: temporary failure
svlogd: fatal: no functional log directories.

---

What's a better way to set this up? How can I have one place to look to see all the log entries from different runit services that are running the same code?

Answer 1

As far as I can tell, svlogd doesn't support combining logs into a single file. However, it does support logging to a socket. My workaround is to instruct svlogd to forward log entries over UDP to the running rsyslog instance.

/etc/sv/test-1/log/run

#!/bin/sh
exec svlogd -tt /var/log/test/1

/var/log/test/1/config

ptest-1: 
U127.0.0.1

/etc/sv/test-2/log/run

#!/bin/sh
exec svlogd -tt /var/log/test/2

/var/log/test/2/config

ptest-2: 
U127.0.0.1

From there, it's just a matter of configuring rsyslog to do what you want.

Answer 2

Use 'logger' to syslog

If you're happy with syslog, then use logger instead of svlogd. The whole point of having a runscript for your logger is that you can run whatever program you like to do the logging.

/etc/sv/test-1/log/run

#!/bin/sh
exec logger -i -p daemon.notice -t my_service

.

Merge when viewed

Another good idea is to just let the logs go to separate places and then use a script that merges the files together as you view them. For example, with the timestamps from svlogd you can just

cat /var/log/test/*/current | sort | less

There are other robust tools out there which can view multiple log files simultaneously, like multitail