I need to monitor a daemon every few minutes with a command and analyse both the exit code and the (multi-line) output.
In case there is a problem I want to be notified one time, but I don't want to keep receiving spam until the problem is fixed, so I also want to suppress the notifications for a few hours. In the meantime, all results are recorded in a database on another server.
I've considered SEC - Simple Event Correlator or a cron job running a batch script, but I see a few problems with either approach.
I have a (mostly) working Bash script that I'm testing with cron, using env variables to keep the time of the last sent notification. This approach got complicated when I couldn't find a way to make the env variables persist between successive runs of the cron job. Maybe I could write the time variables to disk; but I need to set a few env variables and handling files and line substitutions just seems like an ugly and complicated solution.
SEC on the other hand, handles the whole notification suppression bit beautifully, but I can't find a way to have it analyse the exit codes from
spawnactions. Less importantly, it seems a bit cumbersome handling multi-line outputs, needing some convoluted code for something that in bash is fixed with double quotes.
What would be a good solution for this sort of situation? Maybe Ruby/YAML files? I'm not experienced with that. I'm open to alternatives.