2

Is it possible to use wget to download multiple files from a text file and have it save the URL of any failed downloads to a different text file?

I use wget bash scripts to download files from a text file like this:

wget -i "/home/user/downloadURLs.txt"

downloadURLs.txt contains one URL to download per line:

http://[website].com/file1
http://[website].com/file2
http://[website].com/file3
http://[website].com/file4

If one of the files fails to download I'd like to have the failed URL saved to a separate file. But when downloading using wget's -i option, I think the exit code would be either 0 if all downloads succeeded, or an error exit code if any of the downloads failed at all. If I can't get an exit code for each individual URL I can't make it save only the failed URLs.

I think this would work:

#!/bin/bash

#map lines of text file to an array
mapfile -t inputUrls < "/home/user/downloadURLs.txt"

for url in ${inputUrls[@]}
do
    wget "$url"

    if [[ $? != 0 ]]
    "$url" >> "/home/user/failedDownloads.txt"
    fi
done

The reason I'm not sure that I want to do that is because when downloading multiple files from one website, wget will often say "Reusing existing connection to [website]." It seems like that behavior is intended to speed up downloads from one website, and it I assume that optimization is lost if wget is called separately for each URL.

Am I correct in thinking that wget is able to download more efficiently if an input file with -i is used?

If I'm correct, how can I download URLs from a text file and save the failed URLs to a separate file, while still using wget's optimizations? Thanks to anyone for any insight here.

Improve this question

1 Answer 1

Reset to default
2

why not simply redirect the stderr?

$ wget -i test.txt 2> wget-fail.log

$ cat  wget-fail.log 
--2016-11-15 22:06:50--  http://failing-host.com/
Auflösen des Hostnamen »failing-host.com (failing-host.com)«... fehlgeschlagen: Der Name oder der Dienst ist nicht bekannt.
wget: kann die Host-Adresse »failing-host.com« nicht auflösen

[edit]

I do have a logging function set up in the bash script to create a log, which saves both stdout and stderr. Just using 2> seems to create a log file with a similarly huge amount of information in it, including successful downloads. I could approach the problem from the angle of parsing the log file,...

no need for parsing:

$ cat wget.sh 
#!/bin/bash
echo log to stdout
echo >&2 log to stderr
wget -i test.txt 2> wget-fail.log

$ sh wget.sh  &> script.log

$ cat script.log 
log to stdout
log to stderr

$ cat wget-fail.log 
--2016-11-15 23:02:00--  http://failing-host.com/
Auflösen des Hostnamen »failing-host.com (failing-host.com)«... fehlgeschlagen: Der Name oder der Dienst ist nicht bekannt.
wget: kann die Host-Adresse »failing-host.com« nicht auflösen
Improve this answer
3
  • I do have a logging function set up in the bash script to create a log, which saves both stdout and stderr. Just using 2> seems to create a log file with a similarly huge amount of information in it, including successful downloads. I could approach the problem from the angle of parsing the log file, but that seemed like the wrong way to go about it, and figuring out how to parse out just the failed URLs would take me a long time. Do you know of any tricks to parse out just the URLs of the failed downloads easily? Thanks @Timothy
    – InverseTelecine
    Nov 15, 2016 at 21:35
  • Thank you for your help @Timothy. I'm sorry if I'm being dense, but I still don't understand. Your method will save a log with information about all the downloads, successful and failed, but what I tried to say I wanted was a text/log file containing just the failed URL lines and nothing else, like the input file. With your method there's a lot of extra text like download progress and HTTP request sent, awaiting response... 200 OK Length: 180524 (176K) and stuff. Unless I am misunderstanding, which is very possible. I don't know what the script.log part is intended to do in your example
    – InverseTelecine
    Nov 23, 2016 at 1:17
  • @InverseTelecine " I don't know what the script.log part is intended to do in your example " it fullfills your firs replies argument: "I do have a logging function set up in the bash script to create a log,". | " Your method will save a log with information about all the downloads, successful and failed" no, it only logs faild downloads unless the implementation of wget changed meanwhile. " containing just the failed URL lines and nothing else" this is the closesd you can get with standard tooling. You may useawk or sed to shrink wget-fail.log
    – Timothy Truckle
    Nov 23, 2016 at 8:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.