I am maintaining an application that currently consists of 4 processes that are dependant on each other in various ways. At the moment these processes are started, stopped and monitored via a rather "grown" bash script, which contains pearls like these:

# And another rather dirty way to identify the node Api
# process by knowing the filename that is given to node.js.
# Keep in mind not to kill the retrieved process directly, but
# check the running user and the used port to make sure you
# actually got the correct process.
findApiProcessId() {
    local pid=`ps ax | grep node | grep api/server\.js | cut -c -5`

    if [ $pid ]
        echo $pid
        echo -1

The other processes are checked using similar means.

I am now allowed to completely rework this process management, but to be honest I have no real idea where to start. The processes I need to monitor are

  • lighttpd which manages a "stable" pid file
  • mongod which manages a pid file that doesn't seem to be quite reliable. Sometimes it points to a pid that belongs to an entirely different process.
  • At least two node.js instances (running with nohup), which try to do maintain some pid file but fail quite miserably.

I need to be able to start and stop single processes and to query their status, all from the command line. And I need to be able to start that program multiple times in different directories for different "groups" of our application. The child processes currently check the current working directory for configuration files and are able to "choose" their ports without blocking other running instances.

My first thought was to write up a simple process host using Python or C, but I guess that's kind of overkill. So I was looking for an existing tool but searches for "linux process host tool" don't reveal anything useful.

So are there any "standard" process host tools that are able to watch over multiple child processes?

2 Answers 2


How about runit, "a UNIX init scheme with service supervision"?

I think it matches your requirements, i.e.

I cannot answer the "one service in some variants" issue in a clever way, you could of course define the services separately... (there might be some neat symlink-and-examine-my-pwd-solution to this but I'm not sure if trying to be clever here is a good idea; thinking about maintainability)

Edit This ArchWiki page provides a quick overview that might be a better start than runit's page.

  • This seems to come quite close, specifying services seperatly to solve the "one service in some variants" doesn't seem too much of a hassle for me at a first glance. Jan 31, 2013 at 10:13
  • Probably depends on the number of them... and how similar they are. If what differs is (e.g., crude sketch follows) the port and ServerRoot, this could be encoded in the symlinks, i.e. /sv/lighttpd_var-www_80 -> /etc/sv/lighttpd, where the latter inspects its $0 and sets appropriate command line parameters.
    – sr_
    Jan 31, 2013 at 10:23
  • We will be naming them after branches and I am actually quite happy with having seperate configurations for those. But thanks a lot for your effort. Jan 31, 2013 at 11:17

pidof and pgrep will help if you want to script this (and dispense with the dubious ps | lots | of | things idiom). You can also filter by uid, gid, ppid, oldest, newest etc.

The command kill -0 $pid can be used to confirm that a specific process ID exists.

There may be complications if you have multiple instances in different directories that you cannot tell the difference between them via ps. Depending on platform you may be able to distinguish them easily by cwd, e.g. on linux check /proc/$PID/cwd.

# pgrep httpd
# head -1 /usr/local/apache2/logs/httpd.pid
# kill -0 `head -1 /usr/local/apache2/logs/httpd.pid` && echo $?
# ls -l /proc/9483/cwd /proc/9483/exe
lrwxrwxrwx 1 root root 0 2013-01-30 19:40 /proc/9483/cwd -> /
lrwxrwxrwx 1 root root 0 2013-01-30 19:37 /proc/9483/exe -> /usr/local/apache2/bin/httpd

(sorry, Apache's $CWD isn't too interesting...)

Your netstat may also be able to help too:

# netstat -plnt | grep :80
tcp        0      0*               LISTEN     9483/httpd

A commonly used trick in startup scripts used to monitor processes which aren't good about managing a PID file is to start them in a non-daemonizing/non-forking manner as a background job with & (though obviously the program must have a non-daemonizing flag, sometimes referred to as inetd mode), and write $! to a PID file yourself.

pstree is a useful tool for tracking down process hierarchies:

#  pstree -lnp 9483
            |             |-{httpd}(9496)
            |             |-{httpd}(9498)
            |             |-{httpd}(9499)
            |             |-{httpd}(9502)
            |             |-{httpd}(9504)
        [... lots more threads snipped ...] 

As a last resort lsof is the multi-platform tool for tracking things downs by file, connection or PID.

For a standard process supervisor type system check out DJB's daemontools package. His site is http://cr.yp.to/daemontools.html , though frankly the documentation can be a little obtuse, there are plenty of tutorial like pages available.

There are a while load of alternatives (unused by me) listed here: https://serverfault.com/questions/192302/alternative-to-daemontools-djbtools-to-supervise-unix-processes

  • Thanks for pointing out how to make my script a little more maintainable! daemontools is certainly worth a look, thanks for your further links. Jan 31, 2013 at 10:12

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .