0

I'm looking for ideas on how to automatically receive an alert (by email would be nice) if a specific periodic email were not to arrive. I would not necessarily have access to the system that generates the email.

In a recent real world use case I have a summary report that is sent daily to a half dozen or so technicians and managers giving system status of dozens of field locations. We have a second channel that sends alarms, but there are many maintenance issues that can only be discovered by reviewing this report.

Last night I found that the script generating the report had stopped working weeks ago - and no one noticed. Please, before anyone spends much time berating the people who failed to notice (and this would include me) understand that 90% to 100% of the people involved are very good at what they do, and really do care, but are on the border of information overload. I think it is just human nature not to notice when one of dozens of emails that aren't normally critical just happens not to arrive. When it arrives that does seems to trigger us to review it with good results, but no arrival - no mental trigger. We notice and act on emails but fail to notice and act on lack of emails.

My current thought is to send any report emails that I want to monitor to additional "watchdog" email addresses. I would then have independent machines - maybe a couple of Raspberry Pis or VM in the cloud receive the emails and send an alert if too much time goes by without receiving the email. Sort of a watchdog timer for emails.

Maybe something like this already exists and I just don't have the right search terms and someone can point me to the information, but I'd also be interested in any suggestions on how others might approach it. For now I'm heading in a specialized script direction, but would be interested in making it more general purpose.

closed as off-topic by Rui F Ribeiro, schily, Jeff Schaller, msp9011, slm Aug 14 '18 at 20:31

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "Requests for learning materials (tutorials, how-tos etc.) are off topic. The only exception is questions about where to find official documentation (e.g. POSIX specifications). See the Help Center and our Community Meta for more information." – Rui F Ribeiro, schily, msp9011, slm
If this question can be reworded to fit the rules in the help center, please edit the question.

1

Assuming that those tools are available to you, you could cobble something together via procmail and cron. If not, anything that can run shell commands when a specific e-mail arrives should be able to do something very similar.

First, set up a procmail (or whichever other mail filtering tool you use) rule that matches the e-mails you want to monitor, and touch a watchdog file. The c flag causes the rule to process a copy of the e-mail and for processing to continue even if there is a match; that way, the e-mails don't get lost.

:0 c
*^Subject: Regular Report
| /usr/local/bin/regular-report-received.sh

Then write a script to touch a file owned by the user, saving it as /usr/local/bin/regular-report-received.sh in this case:

#!/bin/bash
touch ~/.regular-report-received

(This could be done within the procmail rule, but I like to keep things separate, so there goes.)

Then, you'll need a cron job that regularly checks to make sure the file has been touched recently. For example, you could have, for each user:

 @daily /usr/local/bin/is-regular-report-received.sh

Then there is /usr/local/bin/is-regular-report-received.sh which goes along the lines of

#!/bin/bash
STALE_IF_BEFORE=$(date --date='-36 hours' +%s)
FILE_TIMESTAMP=$(stat -c '%Y' ~/.regular-report-received)
test "${FILE_TIMESTAMP}" -gt "${STALE_IF_BEFORE}" || printf 'Report not received!\n'

With this in place, cron should send each user an e-mail at midnight if they haven't received a report within the past 36 hours.

Adjust to taste.

  • Very good boiler plate. Plus it gives me some assurance that I haven't just overlooked some built-in tool. Sorry, my up votes don't count at this point. I'm planning on handing this outline (and any other quality answers that might come in) over to my student employee and let him take a shot at a generalized "email watchdog" app/device with extra points for it working virtualized and/or on light weight hardware. – Boley Aug 14 '18 at 15:42

Not the answer you're looking for? Browse other questions tagged or ask your own question.