Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I'm aware of wget -i as a way to download a list of URLs. The only trouble is that I need to pass some different POST data to each one, which works for single urls using wget --post-data= but not for lists.

I'm open to any CLI downloader, or even something in JS or Python. I would however like to get either a progress bar for each download or a log file updated each time a dl finishes, or some other way of knowing when a dl finishes.

share|improve this question
    
Each URL has a different POST or they all have the same? –  slm Feb 11 at 2:00
    
Many methods shown here: stackoverflow.com/questions/8634109/… –  slm Feb 11 at 2:06

3 Answers 3

If you already have a list of URLs, just add the POST data to that list. Something like:

www.example.com    postdata1
www.foo.com
www.bar.com       second_post_data

Then, instead of using -i, read the file in a bash loop and pass the data to wget:

while read url post; do wget --post-data="$post" $url; done < list.txt

To run them in parallel so that multiple files are downloaded at the same time, use & instead of ;. Careful though, this will launch a separate wget process for each URL.

while read url post; do wget --post-data="$post" $url & done < list.txt

One trick I use for launching this type of thing is keeping track of how many are currently running and only running the next one if the number is below a threshold, 10 for example:

while read url post; do 
 while [[ "$(pgrep -fc wget)" -gt 9 && $c -gt 10 ]]; do sleep 1; done; 
 ((c++));  
 echo "Launching $url ($c)"; 
 wget --post-data="$post" $url >/dev/null 2>&1 && echo "$c finsihed" & 
done < list.txt

That will launch the first 10 URLs, then wait for one to finish and launch the next.

share|improve this answer

I've not used this tool before but thought I'd add it since it sounds exactly like what you're looking for.

uget

It's a GUI tool called uget. It has the ability to take a input file that would contain the URLs to download along with HTTP POST data from a second file. It's unclear if this allows you to use different POST data per URL or if it's only a single set for all the URLs but might be worth a closer look.

$ uget-gtk --help
...
  -i, --input-file=FILE         add URLs found in FILE.
  --http-post-data=STRING       use the POST method; send STRING as the data.
  --http-post-file=FILE         use the POST method; send contents of FILE
screenshots

ss #1

ss #2

share|improve this answer

You can define a delimited file CSV like with this format (for example) :

URL1;POSTDATA1
URL2;POSTDATA2
...

and parse it using an awk command :

awk -F\; '{print "wget -i "$1" --post-data="$2}' your_CSV_file
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.