Is it possible to use wget to download multiple files from a text file and have it save the URL of any failed downloads to a different text file?
I use wget bash scripts to download files from a text file like this:
wget -i "/home/user/downloadURLs.txt"
downloadURLs.txt contains one URL to download per line:
http://[website].com/file1
http://[website].com/file2
http://[website].com/file3
http://[website].com/file4
If one of the files fails to download I'd like to have the failed URL saved to a separate file. But when downloading using wget's -i
option, I think the exit code would be either 0 if all downloads succeeded, or an error exit code if any of the downloads failed at all. If I can't get an exit code for each individual URL I can't make it save only the failed URLs.
I think this would work:
#!/bin/bash
#map lines of text file to an array
mapfile -t inputUrls < "/home/user/downloadURLs.txt"
for url in ${inputUrls[@]}
do
wget "$url"
if [[ $? != 0 ]]
"$url" >> "/home/user/failedDownloads.txt"
fi
done
The reason I'm not sure that I want to do that is because when downloading multiple files from one website, wget will often say "Reusing existing connection to [website]." It seems like that behavior is intended to speed up downloads from one website, and it I assume that optimization is lost if wget is called separately for each URL.
Am I correct in thinking that wget is able to download more efficiently if an input file with -i
is used?
If I'm correct, how can I download URLs from a text file and save the failed URLs to a separate file, while still using wget's optimizations? Thanks to anyone for any insight here.