0

I have a service that is always supposed to be running. I’d like to know when the service ever stops or gets restarted.

I’ve thought about referencing the service’s PID. If it gets restarted, it would get a new pid. So I would like to send an alert or email whenever the pid changes. What’s the least intrusive way to do this?

Right now I have a cron job writing the pid to a file every 5 minutes. Is there a Linux tool that can monitor this file for pid changes? Or should I have some other thing like a Python script running outside somewhere that can pull this file and monitor that way?

  • Why not just have the cron job read the old value, compare it to the current value, then if it's changed write the new value and send mail? – Stephen Harris Mar 29 at 0:47
0

Vince, for continuous monitoring I recommend that you look into open-source solutions such as Nagios. For your Q, the below BASH script, prog_stat.sh, will write DATE and PID to a file every second, then you can add to the script to parse for the PID change and add follow-on actions. CTRL-C to kill

I have a machine for Nagios dev. This script is for that machine to check Nagios PID once a second. Change frequency of check by changing sleep

    #!/bin/bash
    PROG=Nagios          
    LOG=/home/user/nagios_pid.log   
    echo "" > $LOG             
    while true; do    
    PID=`ps ax | grep -i "$PROG" | awk 'NR==1{print $1}'`                  
    echo `date -u` " "  $PID >> $LOG                      
    sleep 1                                               
    done                 
    exit                   

Script output

    [user ~]#cat nagios_pid.log
    Sat Mar 30 11:15:50 UTC 2019   1171
    Sat Mar 30 11:15:51 UTC 2019   1171
    Sat Mar 30 11:15:52 UTC 2019   1171
    Sat Mar 30 11:15:53 UTC 2019   1171
    Sat Mar 30 11:15:54 UTC 2019   1171
    Sat Mar 30 11:15:55 UTC 2019   1171
    Sat Mar 30 11:15:56 UTC 2019   1261
    Sat Mar 30 11:15:57 UTC 2019   1261
    Sat Mar 30 11:15:58 UTC 2019   1261
    Sat Mar 30 11:15:59 UTC 2019   1261

Sort out for change:

   [user ~]# sort -k 7,7 -u  nagios_pid.log
    Sat Mar 30 11:15:50 UTC 2019   1171
    Sat Mar 30 11:15:56 UTC 2019   1261

Now you see the time when the service was restarted without hand parsing a potentially long file.

-Mark

0

If you system is controlled by systemd, and many modern systems are (type systemctl to check if that's the case), you can configure it to take actions when a service restarts. This answer on ServerFault gives but one example.

Another solution is to wrap your service in a shell script which notifies you when it exits. However, this will not work if the process daemonizes itself.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.