11

If you use brace expansion with wget, you can fetch sequentially-numbered images with ease:

$ wget 'http://www.iqandreas.com/sample-images/100-100-color/'{90..110}'.jpg'

It fetches the first 10 files numbered 90.jpg to 99.jpg just fine, but 100.jpg and onward return a 404: File not found error (I only have 100 images stored on the server). These non-existent files become more of "a problem" if you use a larger range, such as {00..200}, with 100 non-existent files, it increases the script's execution time, and might even become a slight burden (or at least annoyance) on the server.

Is there any way for wget to stop after it has received its first 404 error? (or even better, two in a row, in case there was a missing file in the range for another reason) The answer does not need to use brace expansion; loops are fine too.

  • 1
    In a real-time scenario, you may want to hit every URL to know the status. 1, 2 or even n failures is not the right way when you know [begin .. end] indices. Why would you specify [1..200] range when you know there are only 100 images in [1..100]. I guess you can try GNU parallel for simultaneous requests to speed up the process. – SparKot Jul 22 '14 at 6:55
  • 1
    @SparKotॐ The key is I don't know there are only 100 images on the server, I want the script to download as many images as it can in the series until it has figured out where the end is. – IQAndreas Jul 22 '14 at 7:14
9

If you're happy with a loop:

for url in 'http://www.iqandreas.com/sample-images/100-100-color/'{90..110}'.jpg'
do
    wget "$url" || break
done

That will run wget for each URL in your expansion until it fails, and then break out of the loop.

If you want two failures in a row it gets a bit more complicated:

for url in 'http://www.iqandreas.com/sample-images/100-100-color/'{90..110}'.jpg'
do
    if wget "$url"
    then
        failed=
    elif [ "$failed" ]
    then
        break
    else
        failed=yes
    fi
done

You can shrink that a little with && and || instead of if, but it gets pretty ugly.

I don't believe wget has anything built in to do that.

  • May I suggest using elif to make the second example clearer? Something like this perhaps? gist.github.com/IQAndreas/84cae3f0193b67691ff2 (it only adds one extra line, not including putting the thens on the same line as the ifs) – IQAndreas Jul 22 '14 at 6:33
  • Fair enough. The one-line translation isn't as straightforward now, but it isn't much good anyway. – Michael Homer Jul 22 '14 at 6:37
8

You could use the $? variable to get the return code of wget. If it's non-zero then it means an error occured and you tally it up until it reached a threshold, then it could break out of the loop.

Something like this off the top of my head

#!/bin/bash

threshold=0
for x in {90..110}; do
    wget 'http://www.iqandreas.com/sample-images/100-100-color/'$x'.jpg'
    wgetreturn=$?
    if [[ $wgetreturn -ne 0 ]]; then
        threshold=$(($threshold+$wgetreturn))
        if [[ $threshold -eq 16 ]]; then
                break
        fi
    fi
done

The for loop can be cleaned up a bit, but you can understand the general idea.

Changing the $threshold -eq 16 to -eq 24 would mean it would fail 3 times before it would stop, however it wouldn't be twice in a row, it would be if it failed twice in the loop.

The reason why 16 and 24 are used is that is the total of the return codes.
wget responds with a return code of 8 when it receives a response code that corresponds to an error from the server, and thus 16 is the total after 2 errors.

Stopping when failures only occur twice in a row can be done by resetting the threshold whenever wget succeeds, i.e. when the return code is 0


A list of wget return codes can be found here - http://www.gnu.org/software/wget/manual/html_node/Exit-Status.html

  • Although it can be deduced from the answer, you might want to explicitly point out that a 404 error returns an exit code of 8, hence the magic numbers of 16 and 24. – IQAndreas Jul 22 '14 at 8:23
  • I've updated my answer – Lawrence Jul 22 '14 at 8:26
2

With GNU Parallel this ought to work:

parallel --halt 1 wget ::: 'http://www.iqandreas.com/sample-images/100-100-color/'{90..110}'.jpg'

From version 20140722 you can almost have your "two in a row"-failure: --halt 2% will allow for 2% of the jobs to fail:

parallel --halt 2% wget ::: 'http://www.iqandreas.com/sample-images/100-100-color/'{90..110}'.jpg'
1

IMO, focusing in on wget's exit code/status may be too naive for some use-cases, so here is one that considers the HTTP Status Code as well for some granular decision making.

wget provides a -S/--server-response flag to print out the HTTP Response Headers on STDERR of the command - which we can extract and act upon.

#!/bin/bash

set -eu

error_max=2
error_count=0

urls=( 'http://www.iqandreas.com/sample-images/100-100-color/'{90..110}'.jpg' )

for url in "${urls[@]}"; do
  set +e
  http_status=$( wget --server-response -c "$url" 2>&1 )
  exit_status=$?
  http_status=$( awk '/HTTP\//{ print $2 }' <<<"$http_status" | tail -n 1 )

  if (( http_status >= 400 )); then
    # Considering only HTTP Status errors
    case "$http_status" in
      # Define your actions for each 4XX Status Code below
      410) : Gone
        ;;
      416) : Requested Range Not Satisfiable
        error_count=0  # Reset error_count in case of `wget -c`
        ;;
      403) : Forbidden
        ;&
      404) : Not Found
        ;&
      *)     (( error_count++ ))
        ;;
    esac
  elif (( http_status >= 300 )); then
     # We're unlikely to reach here in case of 1XX, 3XX in $http_status
     # but ..
     exit_status=0
  elif (( http_status >= 200 )); then
     # 2XX in $http_status considered successful
     exit_status=0
  elif (( exit_status > 0 )); then

    # Where wget's exit status is one of
    # 1   Generic error code.
    # 2   Parse error 
    #     - when parsing command-line options, the .wgetrc or .netrc...
    # 3   File I/O error.
    # 4   Network failure.
    # 5   SSL verification failure.
    # 6   Username/password authentication failure.
    # 7   Protocol errors.

    (( error_count++ ))
  fi

  echo "$url -> http_status: $http_status, exit_status=$exit_status, error_count=$error_count" >&2

  if (( error_count >= error_max )); then
    echo "error_count $error_count >= $error_max, bailing out .." >&2
    exit "$exit_status"
  fi

done
-1

In python you can do

from subprocess import *

def main():
    for i in range(90, 110):
       try :
          url = "url/"+str(i)
          check_output(["wget", url])
       except CalledProcessError:
          print "Wget returned none zero output, quiting"
          sys.exit(0)

Checkout the documentation for subprocess if you want to do more https://docs.python.org/2/library/subprocess.html

  • Unless check_output does some magic around wget to detect a 404 - I don't believe there are adequate checks here and so doesn't really answer the question. – shalomb Jun 30 '17 at 0:02
  • It does, read the docs. It checks the output in stdout or stderr. wget has a specific code for 404's – briankip Jul 1 '17 at 8:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.