I have various backup-style scripts which run from cron on a headless server (Ubuntu 14.04), typically on a daily schedule. Cron is configured with a mail server so I get feedback from jobs. Normally these backup scripts are written to run without any stdout/stderr output on success (following the standard Unix paradigm that "no news is good news"), so that they don't clog up my email inbox with lots of junk.
From time to time these fail, and I will get a mail immediately with the stdout/stderr output. However, often these failures are for known reasons, and in particular are transient (i.e. they will likely go away again the next day). For example, my internet connection is a little unreliable, and occasionally remote DNS resolutions will fail (assume that's unfixable for the purposes of this question). Of course, that can't be predicted in advance, so reducing the frequency of the job doesn't work.
What I think I would like is for cron to report back to me only after a particular job has failed for more than n attempts, or after a certain period of time, so I can only get reports of 'permanent' errors that I need to address. Is that possible?
I am using cron 3.0pl1-124ubuntu2 on Ubuntu 14.04, although I'm open to other cron-like software and a more general answer (e.g. a wrapper I can place around my scripts) would be very useful to others, I'm sure.
Options I have considered:
- Incorporate logic into the script itself to handle this - an option, but I was looking for somewhere a bit more generic - some scripts are bash, some python, etc. Also, this would significantly complicate things, as all stdout/stderr the script then did would like need to sit in a wrapper function.
- Use a Continuous Integration server like Jenkins to handle running my jobs - more powerful, probably provides what I'm looking for with various plugins, but significantly more complex to manage, a bit heavyweight (requires a JVM), and not very Unix-y.
- noexcuses - A bit too aggressive, since it would retry remote backups, etc., which will tie up resources and potential cause accidental DoSes. I'd rather the cron job was retried on its original schedule.