Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I've found only puf (Parallel URL fetcher) but could to get it work with reading urls from file and something like

 puf < urls.txt

does not work either.

The operating system installed on the server is ubuntu.

share|improve this question
    
This could be done with Python and pycurl library and a little bit of glue logic in a script. But I don't know of a "canned" tool for that. –  Keith Apr 8 '12 at 8:29
    
@Keith Is this approach better than using some async library as gevent with urllib? –  Moonwalker Apr 8 '12 at 18:37
    
urllib is not designed to be used asynchronously. The libcurl has it's own async loop and can be set up to do at least 1000 simultaneous fetches using the "multi" interface. –  Keith Apr 9 '12 at 5:02
    
@Keith I like your answer best so could you write it as a "real" answer to take due credit for it? –  Moonwalker Apr 10 '12 at 2:21
add comment

migrated from serverfault.com Apr 7 '12 at 16:33

This question came from our site for professional system and network administrators.

6 Answers

up vote 1 down vote accepted

You can implement that using Python and the pycurl library. The pycurl library has the "multi" interface that implements its own even loop that enables multiple simultaneous connections.

However the interface is rather C-like and therefore a bit cumbersome as compared to other, more "Pythonic", code.

I wrote a wrapper for it that builds a more complete browser-like client on top of it. You can use that as an example. See the pycopia.WWW.client module. The HTTPConnectionManager wraps the multi interface.

share|improve this answer
add comment

Using GNU Parallel,

$ parallel -j ${jobs} wget < urls.txt

or xargs from GNU Findutils,

$ xargs -P ${jobs} wget < urls.txt

where ${jobs} is the maximum number of wget you want to allow to run concurrently. Without -j/-P, parallel will run as many jobs at a time as CPU cores (which doesn't necessarily make sense for wget bound by network IO), and xargs will run one at a time.

One nice feature that parallel has over xargs is keeping the output of the concurrently-running jobs separated, but if you don't care about that, xargs is more likely to be pre-installed.

share|improve this answer
add comment

Part of GNU Parallel's man page contains an example of a parallel recursive wget.

https://www.gnu.org/software/parallel/man.html#example__breadth_first_parallel_web_crawler_mirrorer

Otherwise ephemient's answer seems obvious.

share|improve this answer
add comment

The victims of your paralell download won't be amused: they expect one connection to serve each client, setting up several connections means less clients overall. (I.e., this is considered rude behaviour).

share|improve this answer
    
But he might be downloading files from different servers, so this wouldn't apply. –  Renan Jan 16 '13 at 1:01
    
Besides what @vonbrand said, you could get something like "Too many connections" and not be able to download all files. And it can be a bit slower (for example reusing one HTTP connection compared to creating several HTTP connections) –  golimar Feb 6 at 15:10
add comment

One solution would be something like the following:

#!/bin/sh
while IFS= read -r uri; do
    wget -q -- "${uri}" &
done

Then run it as ./puf < urls.txt.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.