I'm aware of wget -i as a way to download a list of URLs. The only trouble is that I need to pass some different POST data to each one, which works for single urls using wget --post-data= but not for lists.

I'm open to any CLI downloader, or even something in JS or Python. I would however like to get either a progress bar for each download or a log file updated each time a dl finishes, or some other way of knowing when a dl finishes.


If you already have a list of URLs, just add the POST data to that list. Something like:

www.example.com    postdata1
www.bar.com       second_post_data

Then, instead of using -i, read the file in a bash loop and pass the data to wget:

while read url post; do wget --post-data="$post" $url; done < list.txt

To run them in parallel so that multiple files are downloaded at the same time, use & instead of ;. Careful though, this will launch a separate wget process for each URL.

while read url post; do wget --post-data="$post" $url & done < list.txt

One trick I use for launching this type of thing is keeping track of how many are currently running and only running the next one if the number is below a threshold, 10 for example:

while read url post; do 
 while [[ "$(pgrep -fc wget)" -gt 9 && $c -gt 10 ]]; do sleep 1; done; 
 echo "Launching $url ($c)"; 
 wget --post-data="$post" $url >/dev/null 2>&1 && echo "$c finsihed" & 
done < list.txt

That will launch the first 10 URLs, then wait for one to finish and launch the next.


You can define a delimited file CSV like with this format (for example) :


and parse it using an awk command :

awk -F\; '{print "wget -i "$1" --post-data="$2}' your_CSV_file

I've not used this tool before but thought I'd add it since it sounds exactly like what you're looking for.


It's a GUI tool called uget. It has the ability to take a input file that would contain the URLs to download along with HTTP POST data from a second file. It's unclear if this allows you to use different POST data per URL or if it's only a single set for all the URLs but might be worth a closer look.

$ uget-gtk --help
  -i, --input-file=FILE         add URLs found in FILE.
  --http-post-data=STRING       use the POST method; send STRING as the data.
  --http-post-file=FILE         use the POST method; send contents of FILE

ss #1

ss #2

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.