2

I have a db of files to download containing fields for filename and download_url with the format:

"foo-1.23.4.jar", "http://example.com/files/12345678/download"
"bar-5.67.8.jar", "http://example.com/files/9876543/download"
"baz-3.31.jar", "http://example.com/files/42424242/download"

Where the urls tend to all be the same except for the number. I tried exporting the list of URLs and downloading these with wget -i, but every file gets named download, and I have no way to tell them apart.

Normally I would use the -O parameter to specify the correct output file, but I'm not sure how to combine that with -i

How can I format the input file and command line to wget -i such that each line can specify both a download url and an output filename?

4

Concurrently with GNU parallel:

parallel -a files_list.txt -j0 -C ", *" 'wget -q -O {1} {2}' 
  • -a input-file - use input-file as input source
  • -j N - run up to N jobs in parallel. 0 means as many as possible
  • -C regex - column separator. The input will be treated as a table with regexp separating the columns. The n'th column can be access using {n} or {n.}. E.g. {3} is the 3rd column
2
  • I like this. I had not heard of parallel. Does this mean that wget does not directly support what I want to do, and the solution is multiple invocations? Jan 23 '18 at 21:40
  • @RyanBemrose, this solution provides you with effective and fast downloading process: you are able to download multiple resources in parallel. Your input file does not fit the wget's -i option because it would expect the file to contain only URLs while your file has a more complex structure Jan 23 '18 at 21:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.