2

I've got some services in /service/... in a gentoo-based system and I'd like to monitor these services. I can check whether they are running or down using svc-status but how can I determine whether they are not running but they should? So they are stopped due to some error or were they terminated. Is it possible at all?

The use case is that I have more instances of the same service but not all of them are running at the same time. So I'd like to get notified by my monitoring script whether, for example, 3 out of 8 instances are running only when I explicitly started 4 instances.

3

This seems like a jobs for Monit.

See How to Install and Setup Monit.

  • Thanks, Monit seems okay. However its too much for my needs. I only want to create a small script for this job. – papaiatis Mar 16 '16 at 14:23
1

You can use the Monit for that.

It will automatically check for services running and will start them if they not. Monit even can email you about that

You'll need to configure for that first, but Monit is a great and lightweight tool for every system administrator.

You can look for few examples how to install and configure Monit here.

1

This discusses my nosh toolset specifically, but some of the concepts apply to other members of the daemontools family.

You might want to tell the Gentoo people that their Process Supervision wiki article is woefully outdated and incomplete.

Service status for humans

Obtaining service status in human-readable form is of course done with the svstat, s6-svstat, sv stat, or perpstat commands.

The nosh toolset has a svstat (a.k.a. service-status) command. This has, like the other daemontools family tools, to be pointed directly at the desired service bundle directories. The toolset also provides a system-control status shim that looks up the service bundle directories (in various conventional places) from just a plain service/target name and invokes service-status.

The nosh toolset's service-status command can print out a one-line-per-service human-readable form or a multiple-line human-readable form. Both contain the service's current state: stopped, starting, started, running, or stopping. In the 1-line form, a note is added to the line if that state differs from its enable/disable state (that specifies whether the service should be initially up or down at bootstrap). In recent versions, colour is used to draw attention to that note.

The multiple-line human-readable form always contains the explicit enable/disable state of a service, alongside several other things such as the tail end of the service's log (if it has a conventional log/ service with a conventional main/ directory). So, as a human, you just read and compare.

But this is human-readable form. It's actually tricky to parse reliably. And that's only taking into account the 1-line forms of human-readable output.

Service status for programs

One of the qmail security maxims, first propounded in the 20th century, holds that parsing human-readable stuff for a program-to-program interface is a bad idea. To implement a monitoring program (e.g. a monitoring script), far better to use a tool that employs a machine-readable interface.

One such machine-readable interface is the service management API itself, of course. The (daemontools-encore compatible) control/status API has been stable and known for many years, and is just FIFOs and ordinary files in the filesystem. One can write program libraries that look directly at the supervise/ok and supervise/status files of a service to obtain status, and people have done so. See Peter Ruibal's and Andrés J. Díaz's supervise library for Python and Voxer's Node JavaScript library, for examples.

The nosh toolset also comes with the svshow (a.k.a. service-show and with a similar system-control show shim) command that explicitly produces machine readable output, in either Microsoft INI or JSON formats, about service status.

Integration into server/datacentre monitoring tools

Various monitoring systems know about the supervise/status system of Bernstein daemontools, as extended by Guenter daemontools-encore and employed by nosh service management. They can be employed as is.

For added convenience, the nosh toolset comes with a command named nagios-check-service that can be employed directly as a Nagios plug-in. It speaks the Nagios plug-in protocol, employing the appropriate exit statuses and writing the appropriate things to standard output/error, and so can be dropped directly in to a command definition in /etc/nagios/nrpe.d/.

Further reading

  • Jonathan de Boyne Pollard (2015). The daemontools family. Frequently Given Answers.
  • Possibly slight out of date nosh Guide:
  • The up-to-date nosh Guide is available as a Debian/Ubuntu package and a FreeBSD/PC-BSD/DragonFlyBSD/&c. package, and the manual is accessible on your machine without any Internet connection required via:
    • man service-status
    • man service-show
    • man nagios-check-service
    • xdg-open /usr/local/share/doc/nosh/service-status.html
    • xdg-open /usr/local/share/doc/nosh/service-show.html
    • xdg-open /usr/local/share/doc/nosh/system-control.html
  • Bruce Guenter. svstat. daemontools-encore manual. §8.
  • Gerrit Pape. sv. runit manual. §8.
  • Wayne Marshall (2013). perpstat. perp manual. §8.
  • Laurent Bercot. s6-svstat. s6 manual. Skarnet software.
0

I you are running systemd, you can query all service status with:

systemctl list-units
  • The questioner is running Bruce Guenter's svc-status against daemontools services living in /service/. This is very clearly not systemd. – JdeBP May 15 '16 at 8:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.