3

I am using wget in a bash script to get a list of files from a website, the files are sequential and in the format http://example.com/00001.csv etc.

Is there a way to terminate/end the script when I hit the last file?

This is the script I have so far.

#!/bin/bash 
url="example.com"; 
for a in {00000001..18923230} do 
wget -nc -q ${url}$a.csv 
done 
  • 1
    Do you have any script? It would be good if you post the script and then ask the question. – Ramesh Mar 7 '14 at 16:24
  • #!/bin/bash url="site.com" for a in {00000001..18923230} do wget -nc -q ${url}$a.csv done – Kevin Wincott Mar 7 '14 at 16:25
  • Add an echo statement after done. – Ramesh Mar 7 '14 at 16:27
  • its going to be run on a cronjob so ideally it needs to terminate when it gets the first http 404 rather than iterating from 1 to 18923230 each time it runs (every 2 hours) – Kevin Wincott Mar 7 '14 at 16:29
6

The simplest way would be to check if the file exists. Using your current script:

#!/bin/bash 
url="example.com"; 
for a in {00000001..18923230}
do 
    wget -nc -q ${url}$a.csv || exit;
done 

That will exit as soon as one of the files is not downloadable. Alternatively, assuming there is an index.html file that links to these, you could download that and grep for the file names.

  • @devnull fair point, I used exit 'cause the OP said it should terminate whet it gets the 1st 404. – terdon Mar 7 '14 at 16:45
2

wget appears to return 0 when successful and non-zero otherwise, so:

    wget blahblah
    if [[ $? -ne "0" ]]; then
        break;
    fi

...inside your for loop.

  • Or you could just do if ! wget blahblah; then... or even better, what terdon did in his comment. – orion Mar 7 '14 at 16:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.