Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

A few years ago, a coworker came up with an elegant solution for a watchdog program. The program ran on Windows and used Windows Event objects to monitor the process handles (PID’s) of several applications. If any one of the processes terminated unexpectedly, its process handle would no longer exist and his watchdog would immediately be signaled. The watchdog would then take an appropriate action to “heal” the system.

My question is, how would you implement such a watchdog on Linux? Is there a way for a single program to monitor the PID’s of many others?

share|improve this question
    
See also Ensure a process is always running for a list of monitoring/supervising programs. –  Gilles Jun 6 '11 at 21:47
    
Also see stackoverflow: stackoverflow.com/questions/9400724/… –  Mike Pennington May 1 '12 at 18:49
add comment

2 Answers

up vote 9 down vote accepted

The traditional, portable, commonly-used way is that the parent process watches over its children.

The basic primitives are the wait and waitpid system calls. When a child process dies, the parent process receives a SIGCHLD signal, telling it it should call wait to know which child died and its exit status. The parent process can instead choose to ignore SIGCHLD and call waitpid(-1, &status, WNOHANG) at its convenience.

To monitor many processes, you would either spawn them all from the same parent, or invoke them all through a simple monitoring process that just calls the desired program, waits for it to terminate and reports on the termination (in shell syntax: myprogram; echo myprogram $? >>/var/run/monitor-collector-pipe). If you're coming from the Windows world, note that having small programs doing one specialized task is a common design in the Unix world, the OS is designed to make processes cheap.

There are many process monitoring (also called supervisor) programs that can report when a process dies and optionally restart it and far more besides: Monit, Supervise, Upstart, …

share|improve this answer
    
+1, exactly what I was looking for - thank you for such a detailed answer. –  Justin Ethier Feb 20 '11 at 2:56
add comment

My approach to this issue is to use init and its built-in respawn directive to start/restart whatever you need to run. This was its original intent and main purpose. In some cases you will need to run a script to cleanup after a process has died, or to prepare for the process to start (most of the time the work is the same). In most cases a bash script that ends in exec works great for this.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.