If you use brace expansion with wget, you can fetch sequentially-numbered images with ease:

$ wget 'http://www.iqandreas.com/sample-images/100-100-color/'{90..110}'.jpg'

It fetches the first 10 files numbered 90.jpg to 99.jpg just fine, but 100.jpg and onward return a 404: File not found error (I only have 100 images stored on the server). These non-existent files become more of "a problem" if you use a larger range, such as {00..200}, with 100 non-existent files, it increases the script's execution time, and might even become a slight burden (or at least annoyance) on the server.

Is there any way for wget to stop after it has received its first 404 error? (or even better, two in a row, in case there was a missing file in the range for another reason) The answer does not need to use brace expansion; loops are fine too.

  • 1
    In a real-time scenario, you may want to hit every URL to know the status. 1, 2 or even n failures is not the right way when you know [begin .. end] indices. Why would you specify [1..200] range when you know there are only 100 images in [1..100]. I guess you can try GNU parallel for simultaneous requests to speed up the process. Commented Jul 22, 2014 at 6:55
  • 1
    @SparKotॐ The key is I don't know there are only 100 images on the server, I want the script to download as many images as it can in the series until it has figured out where the end is.
    – IQAndreas
    Commented Jul 22, 2014 at 7:14

6 Answers 6


If you're happy with a loop:

for url in 'http://www.iqandreas.com/sample-images/100-100-color/'{90..110}'.jpg'
    wget "$url" || break

That will run wget for each URL in your expansion until it fails, and then break out of the loop.

If you want two failures in a row it gets a bit more complicated:

for url in 'http://www.iqandreas.com/sample-images/100-100-color/'{90..110}'.jpg'
    if wget "$url"
    elif [ "$failed" ]

You can shrink that a little with && and || instead of if, but it gets pretty ugly.

I don't believe wget has anything built in to do that.

  • May I suggest using elif to make the second example clearer? Something like this perhaps? gist.github.com/IQAndreas/84cae3f0193b67691ff2 (it only adds one extra line, not including putting the thens on the same line as the ifs)
    – IQAndreas
    Commented Jul 22, 2014 at 6:33
  • Fair enough. The one-line translation isn't as straightforward now, but it isn't much good anyway. Commented Jul 22, 2014 at 6:37

You could use the $? variable to get the return code of wget. If it's non-zero then it means an error occured and you tally it up until it reached a threshold, then it could break out of the loop.

Something like this off the top of my head


for x in {90..110}; do
    wget 'http://www.iqandreas.com/sample-images/100-100-color/'$x'.jpg'
    if [[ $wgetreturn -ne 0 ]]; then
        if [[ $threshold -eq 16 ]]; then

The for loop can be cleaned up a bit, but you can understand the general idea.

Changing the $threshold -eq 16 to -eq 24 would mean it would fail 3 times before it would stop, however it wouldn't be twice in a row, it would be if it failed twice in the loop.

The reason why 16 and 24 are used is that is the total of the return codes.
wget responds with a return code of 8 when it receives a response code that corresponds to an error from the server, and thus 16 is the total after 2 errors.

Stopping when failures only occur twice in a row can be done by resetting the threshold whenever wget succeeds, i.e. when the return code is 0

A list of wget return codes can be found here - http://www.gnu.org/software/wget/manual/html_node/Exit-Status.html

  • 2
    Although it can be deduced from the answer, you might want to explicitly point out that a 404 error returns an exit code of 8, hence the magic numbers of 16 and 24.
    – IQAndreas
    Commented Jul 22, 2014 at 8:23
  • 1
    I've updated my answer
    – Lawrence
    Commented Jul 22, 2014 at 8:26

IMO, focusing in on wget's exit code/status may be too naive for some use-cases, so here is one that considers the HTTP Status Code as well for some granular decision making.

wget provides a -S/--server-response flag to print out the HTTP Response Headers on STDERR of the command - which we can extract and act upon.


set -eu


urls=( 'http://www.iqandreas.com/sample-images/100-100-color/'{90..110}'.jpg' )

for url in "${urls[@]}"; do
  set +e
  http_status=$( wget --server-response -c "$url" 2>&1 )
  http_status=$( awk '/HTTP\//{ print $2 }' <<<"$http_status" | tail -n 1 )

  if (( http_status >= 400 )); then
    # Considering only HTTP Status errors
    case "$http_status" in
      # Define your actions for each 4XX Status Code below
      410) : Gone
      416) : Requested Range Not Satisfiable
        error_count=0  # Reset error_count in case of `wget -c`
      403) : Forbidden
      404) : Not Found
      *)     (( error_count++ ))
  elif (( http_status >= 300 )); then
     # We're unlikely to reach here in case of 1XX, 3XX in $http_status
     # but ..
  elif (( http_status >= 200 )); then
     # 2XX in $http_status considered successful
  elif (( exit_status > 0 )); then

    # Where wget's exit status is one of
    # 1   Generic error code.
    # 2   Parse error 
    #     - when parsing command-line options, the .wgetrc or .netrc...
    # 3   File I/O error.
    # 4   Network failure.
    # 5   SSL verification failure.
    # 6   Username/password authentication failure.
    # 7   Protocol errors.

    (( error_count++ ))

  echo "$url -> http_status: $http_status, exit_status=$exit_status, error_count=$error_count" >&2

  if (( error_count >= error_max )); then
    echo "error_count $error_count >= $error_max, bailing out .." >&2
    exit "$exit_status"


With GNU Parallel this ought to work:

parallel --halt 1 wget ::: 'http://www.iqandreas.com/sample-images/100-100-color/'{90..110}'.jpg'

From version 20140722 you can almost have your "two in a row"-failure: --halt 2% will allow for 2% of the jobs to fail:

parallel --halt 2% wget ::: 'http://www.iqandreas.com/sample-images/100-100-color/'{90..110}'.jpg'

What I've used successfully is

wget 'http://www.iqandreas.com/sample-images/100-100-color/'{90..110}'.jpg' 2>&1 | grep -q 'ERROR 404: Not Found'

grep -q looks for the 404 error message pattern in its input and dies as soon as it sees it. wget receives a SIGPIPE signal as soon as it tries to write to the pipe from which grep is no longer reading. In practice wget dies pretty quickly after getting that first 404.


In python you can do

from subprocess import *

def main():
    for i in range(90, 110):
       try :
          url = "url/"+str(i)
          check_output(["wget", url])
       except CalledProcessError:
          print "Wget returned none zero output, quiting"

Checkout the documentation for subprocess if you want to do more https://docs.python.org/2/library/subprocess.html

  • Unless check_output does some magic around wget to detect a 404 - I don't believe there are adequate checks here and so doesn't really answer the question.
    – shalomb
    Commented Jun 30, 2017 at 0:02
  • It does, read the docs. It checks the output in stdout or stderr. wget has a specific code for 404's
    – briankip
    Commented Jul 1, 2017 at 8:36

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .