Curl parallel download file list

Question

first post sorry if I did some mess here =)

Using Ubuntu 14.04lts 64bits server version.

I have a list (url.list) with only URLs to download, one per line, that looks like this:

http://domain.com/teste.php?a=2&b=3&name=1
http://domain.com/teste.php?a=2&b=3&name=2
...
http://domain.com/teste.php?a=2&b=3&name=30000

As you can see, there are many lines in the file (in this case 30000). Because of that I'm using a trick to download many URLs simultaneosly with this:

cat url.list | xargs -n 1 -P 10 <<MAGIC COMMAND THAT WILL SAVE ME>>

The problem is that I'd like to rename the output file with the same value of the name field, like: 1.html, 2.html, ..., 30000.html ecc, and use curl to limit the size of the file to 50KB. So the curl command should be something like:

curl -r 0-50000 -L $URL -o $filename.html -a $filename.log

How can I have it done?

I can parse the output of the pipe with echo $URL | sed -n -e 's/^.*name=//p' but I don't know how use this in the same line grabbing the output of a pipe in 2 variables ($URL and $filename).

I tried this with no success:

cat url.list | xargs -n 1 -P 10 | filename=$(sed -n -e 's/^.*name=//p') ; curl -r 0-50000 -L $URL -o $filename.html -a $filename.log

Thank you in advance

Answer 1

seq 50000 |
  parallel -P 10 curl -r 0-50000 -L '"http://domain.com/teste.php?a=2&b=3&name="'{} -o {}.html -a {}.log