The package I recommend for this is called
daemontools by Dan Bernstein.
This is a collection of tools to provide system-wide service supervision and to manage services. It not only cares about starting and stopping services, but also supervises the service daemons while they are running. Amongst other things, it provides a reliable interface to send signals to service daemons without the need for pid-files, and a log facility with automatic log file rotation and disk space limits.
It satisfies all of your requirements. It's ultra-reliable, once you set it up and understand how to use it, it requires very little maintenance. If there's a problem in your system, it won't be daemontools.
- All control is via command line.
- The daemons will be restarted on sytem restart.
- The daemons can be stopped, started, stopped, suspended from the CLI.
- Plus, handles logging for each daemon too.
- It's manages fast restarts (when a program dies quickly).
This package and underlying design are rock solid. The source code hasn't changed in years, but don't let that fool you. It hasn't needed to change because it's correct.
I've personally used this package to reliably control hundreds of daemon processes on one machine at a time.
Configuration of a new client is easy, just place a control file in the specified directory and it will be automatically started and restarted forever, unless you intervene. Once you know what the file should look like then you make a template or a way to parameterize the control file creation.
I think your best bet is to get the RPM source package from kteru on github and build your own RPM from it. It's easy to build, but the RPM will make it easier to manage and replicate your system.
The homepage and documentation are located at http://cr.yp.to/daemontools.html
The CentOS 4-7 RPM source package is available on github: https://github.com/kteru/daemontools-rpm
There's also a package called
runit that I think is a branch of
daemontools without some of the licensing and distribution restrictions of
daemontools and a more flexible directory layout policy. It is in the Debian repositories, I don't know about CentOS.