16

If you use brace expansion with wget, you can fetch sequentially-numbered images with ease:

$ wget 'http://www.iqandreas.com/sample-images/100-100-color/'{90..110}'.jpg'

It fetches the first 10 files numbered 90.jpg to 99.jpg just fine, but 100.jpg and onward return a 404: File not found error (I only have 100 images stored on the server). These non-existent files become more of "a problem" if you use a larger range, such as {00..200}, with 100 non-existent files, it increases the script's execution time, and might even become a slight burden (or at least annoyance) on the server.

Is there any way for wget to stop after it has received its first 404 error? (or even better, two in a row, in case there was a missing file in the range for another reason) The answer does not need to use brace expansion; loops are fine too.

2
  • 1
    In a real-time scenario, you may want to hit every URL to know the status. 1, 2 or even n failures is not the right way when you know [begin .. end] indices. Why would you specify [1..200] range when you know there are only 100 images in [1..100]. I guess you can try GNU parallel for simultaneous requests to speed up the process. Commented Jul 22, 2014 at 6:55
  • 1
    @SparKotॐ The key is I don't know there are only 100 images on the server, I want the script to download as many images as it can in the series until it has figured out where the end is.
    – IQAndreas
    Commented Jul 22, 2014 at 7:14

6 Answers 6

12

If you're happy with a loop:

for url in 'http://www.iqandreas.com/sample-images/100-100-color/'{90..110}'.jpg'
do
    wget "$url" || break
done

That will run wget for each URL in your expansion until it fails, and then break out of the loop.

If you want two failures in a row it gets a bit more complicated:

for url in 'http://www.iqandreas.com/sample-images/100-100-color/'{90..110}'.jpg'
do
    if wget "$url"
    then
        failed=
    elif [ "$failed" ]
    then
        break
    else
        failed=yes
    fi
done

You can shrink that a little with && and || instead of if, but it gets pretty ugly.

I don't believe wget has anything built in to do that.

2
  • May I suggest using elif to make the second example clearer? Something like this perhaps? gist.github.com/IQAndreas/84cae3f0193b67691ff2 (it only adds one extra line, not including putting the thens on the same line as the ifs)
    – IQAndreas
    Commented Jul 22, 2014 at 6:33
  • Fair enough. The one-line translation isn't as straightforward now, but it isn't much good anyway. Commented Jul 22, 2014 at 6:37
10

You could use the $? variable to get the return code of wget. If it's non-zero then it means an error occured and you tally it up until it reached a threshold, then it could break out of the loop.

Something like this off the top of my head

#!/bin/bash

threshold=0
for x in {90..110}; do
    wget 'http://www.iqandreas.com/sample-images/100-100-color/'$x'.jpg'
    wgetreturn=$?
    if [[ $wgetreturn -ne 0 ]]; then
        threshold=$(($threshold+$wgetreturn))
        if [[ $threshold -eq 16 ]]; then
                break
        fi
    fi
done

The for loop can be cleaned up a bit, but you can understand the general idea.

Changing the $threshold -eq 16 to -eq 24 would mean it would fail 3 times before it would stop, however it wouldn't be twice in a row, it would be if it failed twice in the loop.

The reason why 16 and 24 are used is that is the total of the return codes.
wget responds with a return code of 8 when it receives a response code that corresponds to an error from the server, and thus 16 is the total after 2 errors.

Stopping when failures only occur twice in a row can be done by resetting the threshold whenever wget succeeds, i.e. when the return code is 0


A list of wget return codes can be found here - http://www.gnu.org/software/wget/manual/html_node/Exit-Status.html

2
  • 2
    Although it can be deduced from the answer, you might want to explicitly point out that a 404 error returns an exit code of 8, hence the magic numbers of 16 and 24.
    – IQAndreas
    Commented Jul 22, 2014 at 8:23
  • 1
    I've updated my answer
    – Lawrence
    Commented Jul 22, 2014 at 8:26
4

IMO, focusing in on wget's exit code/status may be too naive for some use-cases, so here is one that considers the HTTP Status Code as well for some granular decision making.

wget provides a -S/--server-response flag to print out the HTTP Response Headers on STDERR of the command - which we can extract and act upon.

#!/bin/bash

set -eu

error_max=2
error_count=0

urls=( 'http://www.iqandreas.com/sample-images/100-100-color/'{90..110}'.jpg' )

for url in "${urls[@]}"; do
  set +e
  http_status=$( wget --server-response -c "$url" 2>&1 )
  exit_status=$?
  http_status=$( awk '/HTTP\//{ print $2 }' <<<"$http_status" | tail -n 1 )

  if (( http_status >= 400 )); then
    # Considering only HTTP Status errors
    case "$http_status" in
      # Define your actions for each 4XX Status Code below
      410) : Gone
        ;;
      416) : Requested Range Not Satisfiable
        error_count=0  # Reset error_count in case of `wget -c`
        ;;
      403) : Forbidden
        ;&
      404) : Not Found
        ;&
      *)     (( error_count++ ))
        ;;
    esac
  elif (( http_status >= 300 )); then
     # We're unlikely to reach here in case of 1XX, 3XX in $http_status
     # but ..
     exit_status=0
  elif (( http_status >= 200 )); then
     # 2XX in $http_status considered successful
     exit_status=0
  elif (( exit_status > 0 )); then

    # Where wget's exit status is one of
    # 1   Generic error code.
    # 2   Parse error 
    #     - when parsing command-line options, the .wgetrc or .netrc...
    # 3   File I/O error.
    # 4   Network failure.
    # 5   SSL verification failure.
    # 6   Username/password authentication failure.
    # 7   Protocol errors.

    (( error_count++ ))
  fi

  echo "$url -> http_status: $http_status, exit_status=$exit_status, error_count=$error_count" >&2

  if (( error_count >= error_max )); then
    echo "error_count $error_count >= $error_max, bailing out .." >&2
    exit "$exit_status"
  fi

done
2

With GNU Parallel this ought to work:

parallel --halt 1 wget ::: 'http://www.iqandreas.com/sample-images/100-100-color/'{90..110}'.jpg'

From version 20140722 you can almost have your "two in a row"-failure: --halt 2% will allow for 2% of the jobs to fail:

parallel --halt 2% wget ::: 'http://www.iqandreas.com/sample-images/100-100-color/'{90..110}'.jpg'
0

What I've used successfully is

wget 'http://www.iqandreas.com/sample-images/100-100-color/'{90..110}'.jpg' 2>&1 | grep -q 'ERROR 404: Not Found'

grep -q looks for the 404 error message pattern in its input and dies as soon as it sees it. wget receives a SIGPIPE signal as soon as it tries to write to the pipe from which grep is no longer reading. In practice wget dies pretty quickly after getting that first 404.

-1

In python you can do

from subprocess import *

def main():
    for i in range(90, 110):
       try :
          url = "url/"+str(i)
          check_output(["wget", url])
       except CalledProcessError:
          print "Wget returned none zero output, quiting"
          sys.exit(0)

Checkout the documentation for subprocess if you want to do more https://docs.python.org/2/library/subprocess.html

2
  • Unless check_output does some magic around wget to detect a 404 - I don't believe there are adequate checks here and so doesn't really answer the question.
    – shalomb
    Commented Jun 30, 2017 at 0:02
  • It does, read the docs. It checks the output in stdout or stderr. wget has a specific code for 404's
    – briankip
    Commented Jul 1, 2017 at 8:36

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .