2

I have various backup-style scripts which run from cron on a headless server (Ubuntu 14.04), typically on a daily schedule. Cron is configured with a mail server so I get feedback from jobs. Normally these backup scripts are written to run without any stdout/stderr output on success (following the standard Unix paradigm that "no news is good news"), so that they don't clog up my email inbox with lots of junk.

From time to time these fail, and I will get a mail immediately with the stdout/stderr output. However, often these failures are for known reasons, and in particular are transient (i.e. they will likely go away again the next day). For example, my internet connection is a little unreliable, and occasionally remote DNS resolutions will fail (assume that's unfixable for the purposes of this question). Of course, that can't be predicted in advance, so reducing the frequency of the job doesn't work.

What I think I would like is for cron to report back to me only after a particular job has failed for more than n attempts, or after a certain period of time, so I can only get reports of 'permanent' errors that I need to address. Is that possible?

I am using cron 3.0pl1-124ubuntu2 on Ubuntu 14.04, although I'm open to other cron-like software and a more general answer (e.g. a wrapper I can place around my scripts) would be very useful to others, I'm sure.

Options I have considered:

  • Incorporate logic into the script itself to handle this - an option, but I was looking for somewhere a bit more generic - some scripts are bash, some python, etc. Also, this would significantly complicate things, as all stdout/stderr the script then did would like need to sit in a wrapper function.
  • Use a Continuous Integration server like Jenkins to handle running my jobs - more powerful, probably provides what I'm looking for with various plugins, but significantly more complex to manage, a bit heavyweight (requires a JVM), and not very Unix-y.
  • noexcuses - A bit too aggressive, since it would retry remote backups, etc., which will tie up resources and potential cause accidental DoSes. I'd rather the cron job was retried on its original schedule.
2
  • When you say n attempts, do you mean over n days, or that you want it to try re-running the script n times and report if there has been no successes?
    – Centimane
    Oct 7, 2015 at 12:14
  • @Dave I'm flexible on that, as they mean the same thing if you are running it on a once-a-day schedule (e.g. not one success in 5 days is the same thing as running it once a day across 5 days). The best answer would be something that allowed both :) Oct 7, 2015 at 12:15

2 Answers 2

2

Because you want to track failures over many days you essentially need to log the issues, however these "logs" don't have to concern you, and can be stashed away for only the cronjob. A wrapper around your scripts seems like the way to go.

maxAttempts=5

if [ -f ~/.script_fails ];then #determine the number of failures
    failures=$(cat ~/.script_fails)
else
    failures=0
fi

if [ $failures -lt $maxAttempts ];then #determine if failures exceeds max attempts
    ./script.sh > /dev/null #if so, get rid of output
else
    ./script.sh #otherwise keep it
fi

result=$?

if [ $result -eq 0 ];then #increment or remove counter
    rm ~/.script_fails
else
    failures=$failures + 1
    echo $failures > ~/.script_fails
fi

this determines when logging needs to be kept or ignored, fairly straight forward. The file for tracking fail counts is hidden in your home directory, so you would be able to check it if you wanted to, but your mail shouldn't see anything unless the failures get too high.

I would definitely recommend using jenkins though, it makes life a lot easier when configured.

5
  • Dave, OK, thanks, that's a pretty good home-grown solution. Obviously suitable only for one script in its current form (I have several), would need a bit of genericisation, but seems pretty good. Thanks! Oct 7, 2015 at 12:38
  • If this is bash (and it seems like it) then the increment should probably be failures=$((failures + 1))
    – Dani_l
    Oct 7, 2015 at 12:56
  • @AndrewFerrier If you just create an array of the scripts and replace the ./script.sh with a for loop you can achieve multiple scripts pretty easily. If they require args you can wrap the script and args in quotes in the array element.
    – Centimane
    Oct 7, 2015 at 12:57
  • Dave, sure, it's just that they are configured in different places, in different crontabs, on different machines... - so I was looking for a more generic script, ultimately. But this is definitely along the right lines. I'll work on making it more generic. Oct 7, 2015 at 12:58
  • 1
    @AndrewFerrier for something that complex something like jenkins becomes much more valuable, managing them all from a crontab is asking a lot from bash.
    – Centimane
    Oct 7, 2015 at 13:00
0

I have now begun to write a wrapper utility to solve my own problem here, called cromer. It is now working in a basic form. Any contributions/pull requests/issues etc. welcome.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .