Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I've found only puf (Parallel URL fetcher) but could to get it work with reading urls from file and something like

 puf < urls.txt

does not work either.

The operating system installed on the server is ubuntu.

share|improve this question

migrated from serverfault.com Apr 7 '12 at 16:33

This question came from our site for system and network administrators.

This could be done with Python and pycurl library and a little bit of glue logic in a script. But I don't know of a "canned" tool for that. – Keith Apr 8 '12 at 8:29
@Keith Is this approach better than using some async library as gevent with urllib? – Moonwalker Apr 8 '12 at 18:37
urllib is not designed to be used asynchronously. The libcurl has it's own async loop and can be set up to do at least 1000 simultaneous fetches using the "multi" interface. – Keith Apr 9 '12 at 5:02
@Keith I like your answer best so could you write it as a "real" answer to take due credit for it? – Moonwalker Apr 10 '12 at 2:21
up vote 1 down vote accepted

You can implement that using Python and the pycurl library. The pycurl library has the "multi" interface that implements its own even loop that enables multiple simultaneous connections.

However the interface is rather C-like and therefore a bit cumbersome as compared to other, more "Pythonic", code.

I wrote a wrapper for it that builds a more complete browser-like client on top of it. You can use that as an example. See the pycopia.WWW.client module. The HTTPConnectionManager wraps the multi interface.

share|improve this answer

Using GNU Parallel,

$ parallel -j ${jobs} wget < urls.txt

or xargs from GNU Findutils,

$ xargs -n 1 -P ${jobs} wget < urls.txt

where ${jobs} is the maximum number of wget you want to allow to run concurrently (setting -n to 1 to get one wget invocation per line in urls.txt). Without -j/-P, parallel will run as many jobs at a time as CPU cores (which doesn't necessarily make sense for wget bound by network IO), and xargs will run one at a time.

One nice feature that parallel has over xargs is keeping the output of the concurrently-running jobs separated, but if you don't care about that, xargs is more likely to be pre-installed.

share|improve this answer

Part of GNU Parallel's man page contains an example of a parallel recursive wget.


Otherwise ephemient's answer seems obvious.

share|improve this answer

The victims of your paralell download won't be amused: they expect one connection to serve each client, setting up several connections means less clients overall. (I.e., this is considered rude behaviour).

share|improve this answer
But he might be downloading files from different servers, so this wouldn't apply. – Renan Jan 16 '13 at 1:01
Besides what @vonbrand said, you could get something like "Too many connections" and not be able to download all files. And it can be a bit slower (for example reusing one HTTP connection compared to creating several HTTP connections) – golimar Feb 6 '14 at 15:10
As long as you keep the number sane, it's not a big deal. E.g., at the time you wrote this, Firefox was using 15 connections per server when not using persistent connections (they've since switched to only trying persistent connections, which are limited to 6 per server). Other browsers use similar numbers. – derobert Jun 24 '15 at 3:26

One solution would be something like the following:

while IFS= read -r uri; do
    wget -q -- "${uri}" &

Then run it as ./puf < urls.txt.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.