0

wgetis a great tool to download files or webpages. It is not the first timke that I find that the links of a webpage are not updated or wrong. For instance, a web page which has its pictures/files linked with http://websitehttp//website.file.extension. Is there a way to tell wget that, if no content is found, it has to look in the address http//website.file.extension instead of http://websitehttp//website.file.extension?

EDIT: Following @Tigger's comment, I can get exit status but how to ask wget for that specific file that failed to get it on the right link/address?

wget_output=$(wget –limit-rate=200k –no-clobber –convert-links –random-wait -r -p -E "$URL")
if [ $? -ne 0 ]; then
  ...
fi
  • 1
    wget will exit 0 if there are no errors and from 1 to 8 on another kind of error (see exit status in man page). You could use the exit status within a simply shell script to work out when to try a second link. – Tigger Dec 6 '16 at 22:08
1

Here is a simple script that might help you to get started:

#!/bin/sh

# Make sure a URL is passed first
if [ -z "$1" ]
then
    echo "
Pass the full URL to be downloaded. For example:

 ${0##*/} \"http://websitehttp/website.file.extension\"

If that URL fails, then \"http://website.file.extension\"
will be tried automatically.
"
    exit 1
fi

# Attempt download
wget -v "${1}"

# Check for an error and if so, try an alternative download
if [ "$?" != "0" ]
then
    url2=`echo ${1} | cut -d '/' -f 4-`
    # DEBUG echo "[$url2]"
    wget -v "http://${url2}"
fi

echo "Done"
exit 0
| improve this answer | |
  • thanks again. I have tested the script but please note that I am doing a recursive call to an url and therefore I do not know in advance the complete link address of the different webpages. I only know that some links are wrong because the parent "www.website.com" is duplicated in the link to some files in some webpages. – nopeva Dec 6 '16 at 23:44
  • If you are doing a recursive crawl and the links are wrong then there is very little you can do. My understanding is wget can not guess what the correct link should have been :] One option might be to create a wget log file and then parse the errors in the log file (using a script). – Tigger Dec 7 '16 at 0:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.