How to get notified when svcadm restarts a service that died

Question

I'm using svcadm to keep my node.js server running on Illumos, is there a way to get notified if svcadm restarts the server because it crashed? Specifically if the process went down on its own and not when I restart it.

I'd like a script to run so I can send an email or raise an alert.

Answer 1

On Solaris 11, you can have the SMF daemons notify you of service state changes:

• Notifications for SMF Instance State Transitions
• How to Set Up Notification of SMF Transition Events

but I'm not sure if illumos has that feature or not, since it was a later addition to SMF.

On any release with SMF, you could always just put the command to email/notify you in the start methods of your SMF service, if you are using a script for those instead of just calling the program directly from your SMF manifest.

Answer 2

Each service has it's own log. you can check what is the log for your service by doing and as example I'm using ssh:

#svcs -xv ssh
svc:/network/ssh:default (SSH server)
 State: online since Tue Nov 27 11:38:34 2012
   See: man -M /usr/share/man -s 1M sshd
   See: /var/svc/log/network-ssh:default.log
Impact: None.

Now you can have your own script, let's say in crontab, to check for changes on the log file. the second way and I guess it will actually be the best one, you can change your start script that whenever it starts, it will send you automatically a email.

To find out where the start up script resides:

svccfg -s svc:/network/ssh
listprop
...
start/exec                        astring  "/lib/svc/method/sshd start
...

Now just edit the script and before issuing the start you can put a automatic mail notification

vi /lib/svc/method/sshd    

search for the function start and add something like this:

echo "Service ssh has been restarted!" |mailx -s "SSHD" user@mailbox

Now just update to match your environment and service. Not pretty but it works.