Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I am maintaining an application that currently consists of 4 processes that are dependant on each other in various ways. At the moment these processes are started, stopped and monitored via a rather "grown" bash script, which contains pearls like these:

# And another rather dirty way to identify the node Api
# process by knowing the filename that is given to node.js.
# Keep in mind not to kill the retrieved process directly, but
# check the running user and the used port to make sure you
# actually got the correct process.
findApiProcessId() {
    local pid=`ps ax | grep node | grep api/server\.js | cut -c -5`

    if [ $pid ]
    then
        echo $pid
     else
        echo -1
    fi
}

The other processes are checked using similar means.

I am now allowed to completely rework this process management, but to be honest I have no real idea where to start. The processes I need to monitor are

  • lighttpd which manages a "stable" pid file
  • mongod which manages a pid file that doesn't seem to be quite reliable. Sometimes it points to a pid that belongs to an entirely different process.
  • At least two node.js instances (running with nohup), which try to do maintain some pid file but fail quite miserably.

I need to be able to start and stop single processes and to query their status, all from the command line. And I need to be able to start that program multiple times in different directories for different "groups" of our application. The child processes currently check the current working directory for configuration files and are able to "choose" their ports without blocking other running instances.

My first thought was to write up a simple process host using Python or C, but I guess that's kind of overkill. So I was looking for an existing tool but searches for "linux process host tool" don't reveal anything useful.

So are there any "standard" process host tools that are able to watch over multiple child processes?

share|improve this question

2 Answers 2

up vote 2 down vote accepted

How about runit, "a UNIX init scheme with service supervision"?

I think it matches your requirements, i.e.

I cannot answer the "one service in some variants" issue in a clever way, you could of course define the services separately... (there might be some neat symlink-and-examine-my-pwd-solution to this but I'm not sure if trying to be clever here is a good idea; thinking about maintainability)

Edit This ArchWiki page provides a quick overview that might be a better start than runit's page.

share|improve this answer
    
This seems to come quite close, specifying services seperatly to solve the "one service in some variants" doesn't seem too much of a hassle for me at a first glance. –  Marcus Riemer Jan 31 '13 at 10:13
    
Probably depends on the number of them... and how similar they are. If what differs is (e.g., crude sketch follows) the port and ServerRoot, this could be encoded in the symlinks, i.e. /sv/lighttpd_var-www_80 -> /etc/sv/lighttpd, where the latter inspects its $0 and sets appropriate command line parameters. –  sr_ Jan 31 '13 at 10:23
    
We will be naming them after branches and I am actually quite happy with having seperate configurations for those. But thanks a lot for your effort. –  Marcus Riemer Jan 31 '13 at 11:17

pidof and pgrep will help if you want to script this (and dispense with the dubious ps | lots | of | things idiom). You can also filter by uid, gid, ppid, oldest, newest etc.

The command kill -0 $pid can be used to confirm that a specific process ID exists.

There may be complications if you have multiple instances in different directories that you cannot tell the difference between them via ps. Depending on platform you may be able to distinguish them easily by cwd, e.g. on linux check /proc/$PID/cwd.

# pgrep httpd
9483
9492
9493
9497
# head -1 /usr/local/apache2/logs/httpd.pid
9483
# kill -0 `head -1 /usr/local/apache2/logs/httpd.pid` && echo $?
0
# ls -l /proc/9483/cwd /proc/9483/exe
lrwxrwxrwx 1 root root 0 2013-01-30 19:40 /proc/9483/cwd -> /
lrwxrwxrwx 1 root root 0 2013-01-30 19:37 /proc/9483/exe -> /usr/local/apache2/bin/httpd

(sorry, Apache's $CWD isn't too interesting...)

Your netstat may also be able to help too:

# netstat -plnt | grep :80
tcp        0      0 192.168.123.123:80       0.0.0.0:*               LISTEN     9483/httpd

A commonly used trick in startup scripts used to monitor processes which aren't good about managing a PID file is to start them in a non-daemonizing/non-forking manner as a background job with & (though obviously the program must have a non-daemonizing flag, sometimes referred to as inetd mode), and write $! to a PID file yourself.

pstree is a useful tool for tracking down process hierarchies:

#  pstree -lnp 9483
httpd(9483)-+-rotatelogs(9484)
            |-rotatelogs(9485)
            |-rotatelogs(9486)
            |-rotatelogs(9491)
            |-httpd(9492)
            |-httpd(9493)-+-{httpd}(9495)
            |             |-{httpd}(9496)
            |             |-{httpd}(9498)
            |             |-{httpd}(9499)
            |             |-{httpd}(9502)
            |             |-{httpd}(9504)
        [... lots more threads snipped ...] 

As a last resort lsof is the multi-platform tool for tracking things downs by file, connection or PID.

For a standard process supervisor type system check out DJB's daemontools package. His site is http://cr.yp.to/daemontools.html , though frankly the documentation can be a little obtuse, there are plenty of tutorial like pages available.

There are a while load of alternatives (unused by me) listed here: http://serverfault.com/questions/192302/alternative-to-daemontools-djbtools-to-supervise-unix-processes

share|improve this answer
    
Thanks for pointing out how to make my script a little more maintainable! daemontools is certainly worth a look, thanks for your further links. –  Marcus Riemer Jan 31 '13 at 10:12

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.