wget download files from a file list, how to specify names for each of the downloaded file?

Question

When downloading a single file, we can use wget's -O option to specify the file name. When I'm downloading Urls in a file using wget -i filelist.txt (filelist.txt contains list Of Urls I want to Download), how can I construct filelist.txt so that each file is renamed as it is downloaded?

For Ex, if the filelist.txt contains the following content:

--output-document=1.jpg http://images2.example.com/image1.jpg
--output-document=2.jpg http://images2.example.com/image2.jpg

Is it possible to download image1.jpg and save as 1.jpg, image2.jpg as 2.jpg?

---

Edit

I know I can use simple script or text manipulation techniques to create multiple wget commands and each of which downloads a single Url and write to an output file. I haven't tested but this seems slower to me and I want to know if there is any way to download all the files using a single wget process.

Answer 1

You can't do it purely in one invokation of Wget. This stems from the definition of Wget's -O option, which doesn't simply mean name of saved file, but rather is a shell redirection of stdout.

If all the names of the files are different on the server, then you can still do this fairly quickly by downloading all the files in a single invokation of Wget and then using a shell script to rename them.

You're right in that multiple invokations of wget will be slow. The process startup and teardown is one aspect, but needing to make a new HTTP connection to the server for each of the files can add up very quickly

Answer 2

Just loop through the file line by line, read the line contents into variable FOO and use that as wget parameter:

while read FOO; do echo wget $FOO; done < filelist.txt

This tiny bit of time for constructing the command line with parameters won't hurt you, compared with download times. For the server side it makes no difference at all, if the same or different wget processes do the single requests for files, which are done in both scenarios, the tiny bit off Keep-Alive won't make much difference, in my opinon.

By the way: wget is always single threaded, there is no parameter to make it multi-threaded. Perhaps you should give it a try and see how much slower it really is, starting a new wget process for each URL, instead of using -i and hoping it will be faster, so that you do no have to guess how much slower it is, but just see it.
Also you could start several wget processes, similar like it can be seen in https://stackoverflow.com/questions/7577615/parallel-wget-in-bash/11850469#11850469 (just pass more parameters than there), which should speed up everything, for small files.

And you still could rename the files after download (also this is not really the way you asked for) to achieve your goal, if you need to construct a list of target file names anyways.