7

I'm aware of wget -i as a way to download a list of URLs. The only trouble is that I need to pass some different POST data to each one, which works for single urls using wget --post-data= but not for lists.

I'm open to any CLI downloader, or even something in JS or Python. I would however like to get either a progress bar for each download or a log file updated each time a dl finishes, or some other way of knowing when a dl finishes.

2

3 Answers 3

4

If you already have a list of URLs, just add the POST data to that list. Something like:

www.example.com    postdata1
www.foo.com
www.bar.com       second_post_data

Then, instead of using -i, read the file in a bash loop and pass the data to wget:

while read url post; do wget --post-data="$post" $url; done < list.txt

To run them in parallel so that multiple files are downloaded at the same time, use & instead of ;. Careful though, this will launch a separate wget process for each URL.

while read url post; do wget --post-data="$post" $url & done < list.txt

One trick I use for launching this type of thing is keeping track of how many are currently running and only running the next one if the number is below a threshold, 10 for example:

while read url post; do 
 while [[ "$(pgrep -fc wget)" -gt 9 && $c -gt 10 ]]; do sleep 1; done; 
 ((c++));  
 echo "Launching $url ($c)"; 
 wget --post-data="$post" $url >/dev/null 2>&1 && echo "$c finsihed" & 
done < list.txt

That will launch the first 10 URLs, then wait for one to finish and launch the next.

2

You can define a delimited file CSV like with this format (for example) :

URL1;POSTDATA1
URL2;POSTDATA2
...

and parse it using an awk command :

awk -F\; '{print "wget -i "$1" --post-data="$2}' your_CSV_file
2

I've not used this tool before but thought I'd add it since it sounds exactly like what you're looking for.

uget

It's a GUI tool called uget. It has the ability to take a input file that would contain the URLs to download along with HTTP POST data from a second file. It's unclear if this allows you to use different POST data per URL or if it's only a single set for all the URLs but might be worth a closer look.

$ uget-gtk --help
...
  -i, --input-file=FILE         add URLs found in FILE.
  --http-post-data=STRING       use the POST method; send STRING as the data.
  --http-post-file=FILE         use the POST method; send contents of FILE
screenshots

ss #1

ss #2

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .