We are seeing an issue using monit during a reboot or shutdown sequence. First some background of the some of the configuration of monit.
- Monit version is 5.5
- Monit is started from inittab as null::respawn:/usr/bin/monit –Ic /etc/monitrc
- Monit is configured to check services every 30 secs.
- Monit is monitoring certain custom applications and restarts them if they are not running by calling the init script of that particular custom application.
- The applications have their own init script in /etc/init.d/ folder that are started on bootup. Monit starts after these init scripts are run from inittab.
Now to the problem:
During a shutdown/reboot of the system first a command is sent to monit to stop monitoring all services. After this all the kill scripts in /etc/init.d/ are called. For the most part it works but there is a corner case where monit starts an application when the application has already shutdown gracefully, which is wrong. I have tried to capture the problem in the Sequence diagram below.
I looked at monit source code and it looks like when the check for a service fails monit tries to run the command mentioned by forking and executing it. So this forked process can still be in progress when monit receives a unmonitor command. It also looks like monit does not stop processes that it started when it receives this command and immediately returns.
Is there a way to wait till all the child processes that are started by monit are complete?
Any other recommendation to avoid this problem?