I've found only puf (Parallel URL fetcher) but I couldn't get it to read urls from a file; something like

 puf < urls.txt

does not work either.

The operating system installed on the server is Ubuntu.

  • This could be done with Python and pycurl library and a little bit of glue logic in a script. But I don't know of a "canned" tool for that.
    – Keith
    Apr 8, 2012 at 8:29
  • @Keith Is this approach better than using some async library as gevent with urllib?
    – Moonwalker
    Apr 8, 2012 at 18:37
  • urllib is not designed to be used asynchronously. The libcurl has it's own async loop and can be set up to do at least 1000 simultaneous fetches using the "multi" interface.
    – Keith
    Apr 9, 2012 at 5:02
  • @Keith I like your answer best so could you write it as a "real" answer to take due credit for it?
    – Moonwalker
    Apr 10, 2012 at 2:21

6 Answers 6


Using GNU Parallel,

$ parallel -j ${jobs} wget < urls.txt

or xargs from GNU Findutils,

$ xargs -n 1 -P ${jobs} wget < urls.txt

where ${jobs} is the maximum number of wget you want to allow to run concurrently (setting -n to 1 to get one wget invocation per line in urls.txt). Without -j/-P, parallel will run as many jobs at a time as CPU cores (which doesn't necessarily make sense for wget bound by network IO), and xargs will run one at a time.

One nice feature that parallel has over xargs is keeping the output of the concurrently-running jobs separated, but if you don't care about that, xargs is more likely to be pre-installed.

  • Optimal jobs depends on many factors: path latency, path bandwidth, remote server policies, etc.
    – dhchdhd
    Sep 18, 2017 at 9:13
  • Is there a way to use this when using wget recursively on a single page? Jan 11, 2021 at 4:39

aria2 does this.


Example: aria2c http://example.org/mylinux.iso


You can implement that using Python and the pycurl library. The pycurl library has the "multi" interface that implements its own even loop that enables multiple simultaneous connections.

However the interface is rather C-like and therefore a bit cumbersome as compared to other, more "Pythonic", code.

I wrote a wrapper for it that builds a more complete browser-like client on top of it. You can use that as an example. See the pycopia.WWW.client module. The HTTPConnectionManager wraps the multi interface.


This works, and won't local or remote DoS, with proper adjustments:

(bandwidth=5000 jobs=8; \
 parallel      \
   --round     \
   -P $jobs    \
   --nice +5   \
   --delay 2   \
   --pipepart  \
   --cat       \
   -a urls.txt \
     wget                                \
       --limit-rate=$((bandwidth/jobs))k \
       -w 1                              \
       -nv                               \
       -i {}                             \

Part of GNU Parallel's man page contains an example of a parallel recursive wget.


HTML is downloaded twice: Once for extracting links and once for downloading to disk. Other content is only downloaded once.

If you do not need the recursiveness ephemient's answer seems obvious.

  • Just a late FYI that any parallel plus wget "solution" is both inherently inefficient because it requires downloading content twice, slow because of all the multiphase downloading and it's also not nice to sysops whom have to pay for all your wasting of bandwidth because you didn't use an efficient solution.
    – dhchdhd
    Sep 18, 2017 at 8:23

The victims of your paralell download won't be amused: they expect one connection to serve each client, setting up several connections means less clients overall. (I.e., this is considered rude behaviour).

  • 1
    But he might be downloading files from different servers, so this wouldn't apply.
    – Renan
    Jan 16, 2013 at 1:01
  • Besides what @vonbrand said, you could get something like "Too many connections" and not be able to download all files. And it can be a bit slower (for example reusing one HTTP connection compared to creating several HTTP connections)
    – golimar
    Feb 6, 2014 at 15:10
  • 2
    As long as you keep the number sane, it's not a big deal. E.g., at the time you wrote this, Firefox was using 15 connections per server when not using persistent connections (they've since switched to only trying persistent connections, which are limited to 6 per server). Other browsers use similar numbers.
    – derobert
    Jun 24, 2015 at 3:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.