1

first post sorry if I did some mess here =)

Using Ubuntu 14.04lts 64bits server version.

I have a list (url.list) with only URLs to download, one per line, that looks like this:

http://domain.com/teste.php?a=2&b=3&name=1
http://domain.com/teste.php?a=2&b=3&name=2
...
http://domain.com/teste.php?a=2&b=3&name=30000

As you can see, there are many lines in the file (in this case 30000). Because of that I'm using a trick to download many URLs simultaneosly with this:

cat url.list | xargs -n 1 -P 10 <<MAGIC COMMAND THAT WILL SAVE ME>>

The problem is that I'd like to rename the output file with the same value of the name field, like: 1.html, 2.html, ..., 30000.html ecc, and use curl to limit the size of the file to 50KB. So the curl command should be something like:

curl -r 0-50000 -L $URL -o $filename.html -a $filename.log

How can I have it done?

I can parse the output of the pipe with echo $URL | sed -n -e 's/^.*name=//p' but I don't know how use this in the same line grabbing the output of a pipe in 2 variables ($URL and $filename).

I tried this with no success:

cat url.list | xargs -n 1 -P 10 | filename=$(sed -n -e 's/^.*name=//p') ; curl -r 0-50000 -L $URL -o $filename.html -a $filename.log

Thank you in advance

  • 2
    The cat is useless but more importantly, you will saturate your bandwidth if you try to open thousands of concurrent connections; and if they are really all on the same server, the remote server's admins will want to firewall you for good if you saturate their uplink. The value of parallelization here is doubtful, anyway; your downloads are not constrained by your CPU, but by your bandwidth, most likely - you will probably only cause congestion by adding massive parallelism here. – tripleee May 21 '17 at 17:10
  • Thank you @tripleee for your points. Any suggestion or different approach to have it done? – tonispa May 21 '17 at 19:17
  • I solved part of the problem to pass the urls to curl this way: xargs -n 1 -P 10 curl -s -r 0-50000 -O < url.list But I cant give the name to files as I want 1.html, ..., 3000.html – tonispa May 21 '17 at 19:21
  • You can pass an arbitrarily complex command to xargs with xargs sh -c 'do things with your arguments' _ – tripleee May 23 '17 at 5:10
0
seq 50000 |
  parallel -P 10 curl -r 0-50000 -L '"http://domain.com/teste.php?a=2&b=3&name="'{} -o {}.html -a {}.log 

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.