2

I have a cron job that can fail periodically when resources are not available. Waiting awhile and trying again is the best way to handle such failures. What is the best way to do this? Have the failing script reschedule itself using at? Is there a better method? Perhaps something that already has such retry infrastructure in place.

2
  • Do you prefer the solution to be inside your script, or do you prefer something generic that doesn't require you changing the script (but may look like "ugly" hack in crontab)? Also, what happens if the script fails more than once, do you want it to keep retrying indefinately until it succeeds?
    – aviro
    Feb 8, 2022 at 8:25
  • @aviro I can't imagine I am the only person who has such an issue, so I'm looking to see what others have done to address this issue. For now I can use at however I can't help but think something better exists. So specifically, generic would be nice. A retry maximum would also be nice. Existing software would be great.
    – A. Que
    Feb 8, 2022 at 13:37

3 Answers 3

2

Had a need to keep retrying until a service was available, and so built a dedicated tool to do just this.

https://github.com/minfrin/retry

~$ retry --until=success -- false
retry: 'false' returned 1, backing off for 10 seconds and trying again...
retry: 'false' returned 1, backing off for 10 seconds and trying again...
retry: 'false' returned 1, backing off for 10 seconds and trying again...
^C

Available out the box in recent Debian, Ubuntu, and Nix.

1

Schedule a script that runs your job. Let the script retry to run the job until it succeeds or has tried too many times.

This assumes that your job is executed by running some-command and that it fails gracefully by returning a non-zero exit status:

#!/bin/sh

sleeptime=15m # Sleep for 15 minutes after a failed try.
maxtries=8    # 8 * 15 minutes = about 2 hours total of waiting,
              # not counting running and failing.

while ! some-command; do
        maxtries=$(( maxtries - 1 ))
        if [ "$maxtries" -eq 0 ]; then
                echo Failed >&2
                exit 1
        fi

        sleep "$sleeptime" || break
done
2
  • This does seem to be a really good solution. I was looking to see if the functionality of this script was already built into some existing sub-system I didn't know about.
    – A. Que
    Feb 12, 2022 at 21:24
  • @A.Que I doubt it. It's such a small thing to wrap a job in that it's relatively easy to write (it doesn't get very big anyway).
    – Kusalananda
    Feb 12, 2022 at 21:33
0

Rescheduling with at now + 17 minutes will do the trick. Read man at.

In the cron itself you can reschedule it in case the first command failed:

<command> || at now + 17 minutes <command>

The at will reschedule the command only if the first one failed. And of course it will only reschedule it once, if the second run failed as well, it will not rerun.

The only caveat here is that lose some visibility, for instance you won't see the result of the second run in your mails or in the cron logs.

1
  • Yes, at is the solution I use now. Wondering if that is how this is problem is typically handled, or if there is a better solution.
    – A. Que
    Feb 8, 2022 at 13:38

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .