I want to write logic in shell script which will retry it to run again after 15 sec upto 5 times based on "status code=FAIL" if it fails due to some issue.
15 Answers
for i in 1 2 3 4 5; do command && break || sleep 15; done
Replace "command" with your command. This is assuming that "status code=FAIL" means any non-zero return code.
Variations:
Using the {..}
syntax. Works in most shells, but not BusyBox sh
:
for i in {1..5}; do command && break || sleep 15; done
Using seq
and passing along the exit code of the failed command:
for i in $(seq 1 5); do command && s=0 && break || s=$? && sleep 15; done; (exit $s)
Same as above, but skipping sleep 15
after the final fail. Since it's better to only define the maximum number of loops once, this is achieved by sleeping at the start of the loop if i > 1
:
for i in $(seq 1 5); do [ $i -gt 1 ] && sleep 15; command && s=0 && break || s=$?; done; (exit $s)
-
13Just a note, this works because the
&&
is evaluated before the||
because of operator precedence Sep 4, 2015 at 19:50 -
Shellcheck says: "Note that A && B || C is not if-then-else. C may run when A is true. [SC2015]" Dec 29, 2018 at 10:25
-
2@Mausy5043, for this case it does not matter, since
s=0
is true, andbreak
breaks the loop. Mar 2, 2019 at 17:30 -
1super helpful.. my vpn used to disconnect every now and then and it super frustrating to connect it every time.. for now I'm using
for i in {1..20}; do vpnon; done
. there are many issue though, one being Ctrl+C doesn't work, but I'll figure them out later.– KrishnaSep 7, 2021 at 5:53 -
If your vpnon command does not handle ctrl+c, adding "; sleep 3" after "vpnon" gives you three seconds where you can press ctrl+c. Sep 7, 2021 at 7:41
This script uses a counter n
to limit the attempts at the command to five.
If the command is successful, break
ends the loop.
n=0
until [ "$n" -ge 5 ]
do
command && break # substitute your command here
n=$((n+1))
sleep 15
done
-
could you explain about command here?? does it mean command to run the script Jul 11, 2013 at 12:45
-
1"command" is just the name of the command that you want to check the status of. Jul 11, 2013 at 13:00
-
3Worth noting that you can test if n equals five at the end to know if command succeeded or not.– mattdmDec 23, 2015 at 14:20
-
3
function fail {
echo $1 >&2
exit 1
}
function retry {
local n=1
local max=5
local delay=15
while true; do
"$@" && break || {
if [[ $n -lt $max ]]; then
((n++))
echo "Command failed. Attempt $n/$max:"
sleep $delay;
else
fail "The command has failed after $n attempts."
fi
}
done
}
Example:
retry ping invalidserver
produces this output:
ping: unknown host invalidserver
Command failed. Attempt 2/5:
ping: unknown host invalidserver
Command failed. Attempt 3/5:
ping: unknown host invalidserver
Command failed. Attempt 4/5:
ping: unknown host invalidserver
Command failed. Attempt 5/5:
ping: unknown host invalidserver
The command 'ping invalidserver' failed after 5 attempts
For a real-world, working example with complex commands, see this script.
GNU Parallel has --retries
:
parallel --retries 5 --delay 15s ::: ./do_thing.sh
Example:
parallel -t --retries 5 --delay 0.1s 'echo {};exit {}' ::: {0..10}
-
This doesn't work. --retries is for retries on different machines: "If a job fails, retry it on another computer on which it has not failed. Do this n times. If there are fewer than n computers in --sshlogin GNU parallel will re-use all the computers. This is useful if some jobs fail for no apparent reason (such as network failure)" May 8, 2020 at 16:31
-
4Have you tested?
--retries
is both for local and remote jobs. But for remote jobs GNU Parallel tries to retry the job on another server if possible (maybe this job and that server just do not like eachother for some unknown reason). May 8, 2020 at 16:45 -
Turns out I was confused by the documentation and by one undocumented feature (at least I didn't see doc) of retries - if you fail, and you have retries turned on, you don't see stderr until the last fatal error. May 8, 2020 at 18:07
-
2The --help of parallel could be better. In general it's a nicely developed tool, certainly superior to the alternatives with a few minor downsides. (At least since the citation nagscreen is gone, that was a no-go for me.) Btw: --delay is not delaying only for retries, it delays everything. afaik there is no functionality yet to delay only on error (would be useful– JohnJan 7, 2021 at 4:21
-
@john You should feel free to submit at bug report with a better --help text. savannah.gnu.org/bugs/?func=additem&group=parallel Version 20201222 has
--delay 123auto
which will start out at 123, but adjust the delay up and down depending on whether jobs succeed or fail. Jan 7, 2021 at 12:58
Here is function for retry
function retry()
{
local n=0
local try=$1
local cmd="${@: 2}"
[[ $# -le 1 ]] && {
echo "Usage $0 <retry_number> <Command>"; }
until [[ $n -ge $try ]]
do
$cmd && break || {
echo "Command Fail.."
((n++))
echo "retry $n ::"
sleep 1;
}
done
}
retry $*
Output :
[test@Nagios ~]$ ./retry.sh 3 ping -c1 localhost
PING localhost (127.0.0.1) 56(84) bytes of data.
64 bytes from localhost (127.0.0.1): icmp_seq=1 ttl=64 time=0.207 ms
--- localhost ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.207/0.207/0.207/0.000 ms
[test@Nagios ~]$ ./retry.sh 3 ping -c1 localhostlasjflasd
ping: unknown host localhostlasjflasd
Command Fail..
retry 1 ::
ping: unknown host localhostlasjflasd
Command Fail..
retry 2 ::
ping: unknown host localhostlasjflasd
Command Fail..
retry 3 ::
-
I copy pasted your code in a new file called retry.sh and added a line #!/bin/bash at the top. While running with your given commands in the explanation I don't see anything just prompt comes again. Aug 19, 2013 at 10:54
-
-
-
Sorry, I was bizy.., I have tested again, it's working , check output paste.ubuntu.com/6002711 Aug 19, 2013 at 12:35
-
1this is the most elegant answer in here so far--if you're doing something non-trivial. Thanks for taking the time. May 8, 2019 at 17:42
Here is my favorite one line alias / script
alias retry='while [ $? -ne 0 ] ; do fc -s ; done'
Then you can do stuff like:
$ ps -ef | grep "Next Process"
$ retry
and it will keep running the prior command until it finds "Next Process"
-
3
-
Very cool oneliner! @RicardoStuven thanks for
fc -e "#"
- it works in bash too. Apr 28, 2020 at 10:15 -
1its cool but it would be better to have a pause between the tries. Otherwise it could try thousands of times and be very cpu intensive.– The FoolOct 10, 2021 at 8:23
Having a need to do this multiple times, the scripting was getting out of hand, so I created a dedicated tool for this called retry.
retry --until=success --times=5 --delay=15 command ...
If you need multiple commands, you can use sh -c
, e.g.
retry -- sh -c 'date && false'
Retry is available here: https://github.com/minfrin/retry
-
I like how the tool is conscious that it might be run in a pipeline that expects the command to run just once. Well done. May 7, 2020 at 13:21
-
2
I use this script that makes the retries of a given command, the benefit of this script is that if fails all retries it will preserve the exit code.
#!/usr/bin/env bash
if [ $# -ne 3 ]; then
echo 'usage: retry <num retries> <wait retry secs> "<command>"'
exit 1
fi
retries=$1
wait_retry=$2
command=$3
for i in `seq 1 $retries`; do
echo "$command"
$command
ret_value=$?
[ $ret_value -eq 0 ] && break
echo "> failed with $ret_value, waiting to retry..."
sleep $wait_retry
done
exit $ret_value
Probably it can get simplier
You can use the loop
command, available here, like so:
$ loop './do_thing.sh' --every 15s --until-success --num 5
Which will do your thing every 15 seconds until it succeeds, for a maximum of five times.
See below Example :
n=0
while :
do
nc -vzw1 localhost 3859
[[ $? = 0 ]] && break || ((n++))
(( n >= 5 )) && break
done
I'm trying to connect port 3389 on localhost, it will retry until 5 times fail , if success then it will break the loop.
$?
it's exist status of command if it zero means command successfully run , if other than zero means command fai
Seems little bit complicated, may be someone do it better than this.
-
-
-
$?
it's exist status of command if it zero means command successfully run , if other than zero means command fail Jul 11, 2013 at 7:34 -
is it required to give host and port address. can we do it by giving script location dir only. Jul 11, 2013 at 7:43
-
slight mods for one-liner that has an increasing delay in seconds for the next execution
DELAYS=(0 1 3 5); (for i in 1 2 3 4; do sleep ${DELAYS[$i]}; <COMMAND> && break || [ $i -lt 4 ] && echo "retry in ${DELAYS[$i+1]}s"; done)
example (git push used for the error)
✗ DELAYS=(0 1 3 5); (for i in 1 2 3 4; do sleep ${DELAYS[$i]}; git pish && break || [ $i -lt 4 ] && echo "retry in ${DELAYS[$i+1]}s"; done)
git: 'pish' is not a git command. See 'git --help'.
The most similar command is
push
retry in 1s
git: 'pish' is not a git command. See 'git --help'.
The most similar command is
push
retry in 3s
git: 'pish' is not a git command. See 'git --help'.
The most similar command is
push
retry in 5s
git: 'pish' is not a git command. See 'git --help'.
The most similar command is
push
replace git pish
with proper git push
✗ DELAYS=(0 1 3 5); (for i in 1 2 3 4; do sleep ${DELAYS[$i]}; git push && break || [ $i -lt 4 ] && echo "retry in ${DELAYS[$i+1]}s"; done)
Everything up-to-date
Here's a recursive retry
function for functional programming purists:
retry() {
cmd=$1
try=${2:-15} # 15 by default
sleep_time=${3:-3} # 3 seconds by default
# Show help if a command to retry is not specified.
[ -z "$1" ] && echo 'Usage: retry cmd [try=15 sleep_time=3]' && return 1
# The unsuccessful recursion termination condition (if no retries left)
[ $try -lt 1 ] && echo 'All retries failed.' && return 1
# The successful recursion termination condition (if the function succeeded)
$cmd && return 0
echo "Execution of '$cmd' failed."
# Inform that all is not lost if at least one more retry is available.
# $attempts include current try, so tries left is $attempts-1.
if [ $((try-1)) -gt 0 ]; then
echo "There are still $((try-1)) retrie(s) left."
echo "Waiting for $sleep_time seconds..." && sleep $sleep_time
fi
# Recurse
retry $cmd $((try-1)) $sleep_time
}
Pass it a command (or a function name) and optionally a number of retries and a sleep duration between retries, like so:
retry some_command_or_fn 5 15 # 5 tries, sleep 15 seconds between each
-
This doesn't work for commands more than one word long: cmd="echo blah blah" ... line 10: [: blah: integer expression expected ... Neither does it work for pipes, etc. Mar 14, 2019 at 18:05
-
I don't think any functional programming purist will touch bash, just saying...– qwrMay 9, 2022 at 0:38
Answering this question as existing answers fail to,
- Doesn't throw Error Code.
- By doing
exit errCode
, Bash doesn't honor certain traps such astrap somefunc ERR
COMMAND="SOMECOMMAND"
TOTAL_RETRIES=3
retrycount=0
until [ $retrycount -ge $((TOTAL_RETRIES-1)) ]
do
$COMMAND && break
retrycount=$((retrycount+1))
sleep 1
done
if [ $retrycount -eq $((TOTAL_RETRIES-1)) ]
then
$COMMAND
fi
# Retries a given command given number of times and outputs to given variable
# $1 : Command to be passed : handles both simple, piped commands
# $2 : Final output of the command(if successfull)
# $3 : Number of retrial attempts[Default 5]
function retry_function() {
echo "Command to be executed : $1"
echo "Final output variable : $2"
echo "Total trials [Default:5] : $3"
counter=${3:-5}
local _my_output_=$2 #make sure passed variable is not same as this
i=1
while [ $i -le $counter ]; do
local my_result=$(eval "$1")
# this tests if output variable is populated and accordingly retries,
# Not possible to provide error status/logs(STDIN,STDERR)-owing to subshell execution of command
# if error logs are needed, execute the same code, outside function in same shell
if test -z "$my_result"
then
echo "Trial[$i/$counter]: Execution failed"
else
echo "Trial[$i/$counter]: Successfull execution"
eval $_my_output_="'$my_result'"
break
fi
let i+=1
done
}
retry_function "ping -c 4 google.com | grep \"min/avg/max\" | awk -F\"/\" '{print \$5}'" avg_rtt_time
echo $avg_rtt_time
- To pass in a lengthy command, pass a method echoing the content. Take care of method expansion accordingly in a subshell at appropriate place.
- Wait time can be added too - just before the increment!
- For a complex command, youll have to take care of stringifying it(Good luck)
This is an old question but I found myself returning to this often. My use case was to have a one liner that can retry a command up to n
times that can be used on with kubernetes pod (of course if it will work for bash script).
TRY=6; until [ $TRY -eq 0 ] || <your command over here> ; do echo $TRY; echo "<output message>"; TRY=$(expr $TRY - 1); sleep 15; done;
The one liner is a bit hard to get your head around, but it can be very helpful.