first post sorry if I did some mess here =)
Using Ubuntu 14.04lts 64bits server version.
I have a list (url.list) with only URLs to download, one per line, that looks like this:
http://domain.com/teste.php?a=2&b=3&name=1
http://domain.com/teste.php?a=2&b=3&name=2
...
http://domain.com/teste.php?a=2&b=3&name=30000
As you can see, there are many lines in the file (in this case 30000). Because of that I'm using a trick to download many URLs simultaneosly with this:
cat url.list | xargs -n 1 -P 10 <<MAGIC COMMAND THAT WILL SAVE ME>>
The problem is that I'd like to rename the output file with the same value of the name field, like: 1.html, 2.html, ..., 30000.html ecc, and use curl to limit the size of the file to 50KB. So the curl command should be something like:
curl -r 0-50000 -L $URL -o $filename.html -a $filename.log
How can I have it done?
I can parse the output of the pipe with echo $URL | sed -n -e 's/^.*name=//p' but I don't know how use this in the same line grabbing the output of a pipe in 2 variables ($URL and $filename).
I tried this with no success:
cat url.list | xargs -n 1 -P 10 | filename=$(sed -n -e 's/^.*name=//p') ; curl -r 0-50000 -L $URL -o $filename.html -a $filename.log
Thank you in advance
catis useless but more importantly, you will saturate your bandwidth if you try to open thousands of concurrent connections; and if they are really all on the same server, the remote server's admins will want to firewall you for good if you saturate their uplink. The value of parallelization here is doubtful, anyway; your downloads are not constrained by your CPU, but by your bandwidth, most likely - you will probably only cause congestion by adding massive parallelism here. – tripleee May 21 '17 at 17:10xargs -n 1 -P 10 curl -s -r 0-50000 -O < url.listBut I cant give the name to files as I want 1.html, ..., 3000.html – tonispa May 21 '17 at 19:21xargswithxargs sh -c 'do things with your arguments' _– tripleee May 23 '17 at 5:10