Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

If you use brace expansion with wget, you can fetch sequentially-numbered images with ease:

$ wget 'http://www.iqandreas.com/sample-images/100-100-color/'{90..110}'.jpg'

It fetches the first 10 files numbered 90.jpg to 99.jpg just fine, but 100.jpg and onward return a 404: File not found error (I only have 100 images stored on the server). These non-existent files become more of "a problem" if you use a larger range, such as {00..200}, with 100 non-existent files, it increases the script's execution time, and might even become a slight burden (or at least annoyance) on the server.

Is there any way for wget to stop after it has received its first 404 error? (or even better, two in a row, in case there was a missing file in the range for another reason) The answer does not need to use brace expansion; loops are fine too.

share|improve this question
1  
In a real-time scenario, you may want to hit every URL to know the status. 1, 2 or even n failures is not the right way when you know [begin .. end] indices. Why would you specify [1..200] range when you know there are only 100 images in [1..100]. I guess you can try GNU parallel for simultaneous requests to speed up the process. –  SparKot ॐ Jul 22 at 6:55
    
@SparKotॐ The key is I don't know there are only 100 images on the server, I want the script to download as many images as it can in the series until it has figured out where the end is. –  IQAndreas Jul 22 at 7:14

3 Answers 3

If you're happy with a loop:

for url in 'http://www.iqandreas.com/sample-images/100-100-color/'{90..110}'.jpg'
do
    wget "$url" || break
done

That will run wget for each URL in your expansion until it fails, and then break out of the loop.

If you want two failures in a row it gets a bit more complicated:

for url in 'http://www.iqandreas.com/sample-images/100-100-color/'{90..110}'.jpg'
do
    if wget "$url"
    then
        failed=
    elif [ "$failed" ]
    then
        break
    else
        failed=yes
    fi
done

You can shrink that a little with && and || instead of if, but it gets pretty ugly.

I don't believe wget has anything built in to do that.

share|improve this answer
    
May I suggest using elif to make the second example clearer? Something like this perhaps? gist.github.com/IQAndreas/84cae3f0193b67691ff2 (it only adds one extra line, not including putting the thens on the same line as the ifs) –  IQAndreas Jul 22 at 6:33
    
Fair enough. The one-line translation isn't as straightforward now, but it isn't much good anyway. –  Michael Homer Jul 22 at 6:37

You could use the $? variable to get the return code of wget. If it's non-zero then it means an error occured and you tally it up until it reached a threshold, then it could break out of the loop.

Something like this off the top of my head

#!/bin/bash

threshold=0
for x in {90..110}; do
    wget 'http://www.iqandreas.com/sample-images/100-100-color/'$x'.jpg'
    wgetreturn=$?
    if [[ $wgetreturn -ne 0 ]]; then
        threshold=$(($threshold+$wgetreturn))
        if [[ $threshold -eq 16 ]]; then
                break
        fi
    fi
done

The for loop can be cleaned up a bit, but you can understand the general idea.

Changing the $threshold -eq 16 to -eq 24 would mean it would fail 3 times before it would stop, however it wouldn't be twice in a row, it would be if it failed twice in the loop.

The reason why 16 and 24 are used is that is the total of the return codes.
wget responds with a return code of 8 when it receives a response code that corresponds to an error from the server, and thus 16 is the total after 2 errors.

Stopping when failures only occur twice in a row can be done by resetting the threshold whenever wget succeeds, i.e. when the return code is 0


A list of wget return codes can be found here - http://www.gnu.org/software/wget/manual/html_node/Exit-Status.html

share|improve this answer
    
Although it can be deduced from the answer, you might want to explicitly point out that a 404 error returns an exit code of 8, hence the magic numbers of 16 and 24. –  IQAndreas Jul 22 at 8:23
    
I've updated my answer –  Lawrence Jul 22 at 8:26

With GNU Parallel this ought to work:

parallel --halt 1 wget ::: 'http://www.iqandreas.com/sample-images/100-100-color/'{90..110}'.jpg'

From version 20140722 you can almost have your "two in a row"-failure: --halt 2% will allow for 2% of the jobs to fail:

parallel --halt 2% wget ::: 'http://www.iqandreas.com/sample-images/100-100-color/'{90..110}'.jpg'
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.