How to monitor cron maintenance scripts?

Question

My home server runs a couple of shell scripts regularly for maintenance tasks - mostly backup, but also other stuff. I would like to be alerted in case anything fails but also keep a log of when it works.

Currently my setup looks like this:

• Cron calls one shell script which calls other scripts (just so the one won't get too complex). I decided to use one script with many tasks instead of individual cron items as I don't know how long each will take and I don't want them to interfere with one another.
• My cron setup contains a MAILTO line. I never get any errors.
• I don't have any logging. I just check from time to time whether the backup actually exists.

I know, I could implement into each script the functionality to log to a file (or syslog). Is there a way to define this from a central point so that I do not have to code this into every script individually?

Not sure how to achieve a better monitoring. I think a log analyzer system would be too much for this. Someone suggested running the scripts through Jenkins instead of shell/cron, but that seems to be even more effort.

What is a simple and good option?

Answer 1

I have implemented the following:

• Enabled output to stdout for various steps or added custom output, e.g.:
  • echo "Starting backup..."
  • rsync whatever && echo "Backup successful" || echo "Backup failed"
• Checking the return codes of each step of the script, either exiting the sub-script immediately or continuing, returning an error code at the end of the script
• wrote a wrapper for my maintenance script which redirects all the outputs to a log file and if there are any errors within the maintenance script, I get a mail.

Example of maintenance script (does not exit if any individual step breaks, but returns an error in the end):

#!/bin/bash

RETURNCODE=0

echo "Execution started $(date)"

/root/do_something.sh || RETURNCODE=1

# (...)

exit $RETURNCODE

Example of wrapper script that calls the other script, this one is now in my crontab:

#!/bin/bash

# exit on any error (there should not be any in this script)
set -e

LOGFILE="/var/log/my.log"

# redirect STDOUT and STDERR to logfile...
if /root/maintenance.sh > $LOGFILE 2>&1; then
    # the colon ":" means: do nothing
    :
else
    # on error, send me an email
    mail -s "maintenance script failed" email@address.com < "$LOGFILE"
fi