first post sorry if I did some mess here =)

Using Ubuntu 14.04lts 64bits server version.

I have a list (url.list) with only URLs to download, one per line, that looks like this:


As you can see, there are many lines in the file (in this case 30000). Because of that I'm using a trick to download many URLs simultaneosly with this:

cat url.list | xargs -n 1 -P 10 <<MAGIC COMMAND THAT WILL SAVE ME>>

The problem is that I'd like to rename the output file with the same value of the name field, like: 1.html, 2.html, ..., 30000.html ecc, and use curl to limit the size of the file to 50KB. So the curl command should be something like:

curl -r 0-50000 -L $URL -o $filename.html -a $filename.log

How can I have it done?

I can parse the output of the pipe with echo $URL | sed -n -e 's/^.*name=//p' but I don't know how use this in the same line grabbing the output of a pipe in 2 variables ($URL and $filename).

I tried this with no success:

cat url.list | xargs -n 1 -P 10 | filename=$(sed -n -e 's/^.*name=//p') ; curl -r 0-50000 -L $URL -o $filename.html -a $filename.log

Thank you in advance

  • 2
    The cat is useless but more importantly, you will saturate your bandwidth if you try to open thousands of concurrent connections; and if they are really all on the same server, the remote server's admins will want to firewall you for good if you saturate their uplink. The value of parallelization here is doubtful, anyway; your downloads are not constrained by your CPU, but by your bandwidth, most likely - you will probably only cause congestion by adding massive parallelism here.
    – tripleee
    Commented May 21, 2017 at 17:10
  • Thank you @tripleee for your points. Any suggestion or different approach to have it done?
    – tonispa
    Commented May 21, 2017 at 19:17
  • I solved part of the problem to pass the urls to curl this way: xargs -n 1 -P 10 curl -s -r 0-50000 -O < url.list But I cant give the name to files as I want 1.html, ..., 3000.html
    – tonispa
    Commented May 21, 2017 at 19:21
  • You can pass an arbitrarily complex command to xargs with xargs sh -c 'do things with your arguments' _
    – tripleee
    Commented May 23, 2017 at 5:10

1 Answer 1

seq 50000 |
  parallel -P 10 curl -r 0-50000 -L '"http://domain.com/teste.php?a=2&b=3&name="'{} -o {}.html -a {}.log 

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .