16

I've found only puf (Parallel URL fetcher) but I couldn't get it to read urls from a file; something like 

 puf < urls.txt

does not work either.

The operating system installed on the server is Ubuntu.

| improve this question | |
  • This could be done with Python and pycurl library and a little bit of glue logic in a script. But I don't know of a "canned" tool for that. – Keith Apr 8 '12 at 8:29
  • @Keith Is this approach better than using some async library as gevent with urllib? – Moonwalker Apr 8 '12 at 18:37
  • urllib is not designed to be used asynchronously. The libcurl has it's own async loop and can be set up to do at least 1000 simultaneous fetches using the "multi" interface. – Keith Apr 9 '12 at 5:02
  • @Keith I like your answer best so could you write it as a "real" answer to take due credit for it? – Moonwalker Apr 10 '12 at 2:21
26

Using GNU Parallel,

$ parallel -j ${jobs} wget < urls.txt

or xargs from GNU Findutils,

$ xargs -n 1 -P ${jobs} wget < urls.txt

where ${jobs} is the maximum number of wget you want to allow to run concurrently (setting -n to 1 to get one wget invocation per line in urls.txt). Without -j/-P, parallel will run as many jobs at a time as CPU cores (which doesn't necessarily make sense for wget bound by network IO), and xargs will run one at a time.

One nice feature that parallel has over xargs is keeping the output of the concurrently-running jobs separated, but if you don't care about that, xargs is more likely to be pre-installed.

| improve this answer | |
  • Optimal jobs depends on many factors: path latency, path bandwidth, remote server policies, etc. – dhchdhd Sep 18 '17 at 9:13
6

aria2 does this.

http://sourceforge.net/apps/trac/aria2/wiki/UsageExample#Downloadfileslistedinafileconcurrently

Example: aria2c http://example.org/mylinux.iso

| improve this answer | |
2

You can implement that using Python and the pycurl library. The pycurl library has the "multi" interface that implements its own even loop that enables multiple simultaneous connections.

However the interface is rather C-like and therefore a bit cumbersome as compared to other, more "Pythonic", code.

I wrote a wrapper for it that builds a more complete browser-like client on top of it. You can use that as an example. See the pycopia.WWW.client module. The HTTPConnectionManager wraps the multi interface.

| improve this answer | |
2

This works, and won't local or remote DoS, with proper adjustments:

(bandwidth=5000 jobs=8; \
 parallel      \
   --round     \
   -P $jobs    \
   --nice +5   \
   --delay 2   \
   --pipepart  \
   --cat       \
   -a urls.txt \
     wget                                \
       --limit-rate=$((bandwidth/jobs))k \
       -w 1                              \
       -nv                               \
       -i {}                             \
)
| improve this answer | |
1

Part of GNU Parallel's man page contains an example of a parallel recursive wget.

https://www.gnu.org/software/parallel/man.html#EXAMPLE:-Breadth-first-parallel-web-crawler-mirrorer

HTML is downloaded twice: Once for extracting links and once for downloading to disk. Other content is only downloaded once.

If you do not need the recursiveness ephemient's answer seems obvious.

| improve this answer | |
  • Just a late FYI that any parallel plus wget "solution" is both inherently inefficient because it requires downloading content twice, slow because of all the multiphase downloading and it's also not nice to sysops whom have to pay for all your wasting of bandwidth because you didn't use an efficient solution. – dhchdhd Sep 18 '17 at 8:23
0

The victims of your paralell download won't be amused: they expect one connection to serve each client, setting up several connections means less clients overall. (I.e., this is considered rude behaviour).

| improve this answer | |
  • 1
    But he might be downloading files from different servers, so this wouldn't apply. – Renan Jan 16 '13 at 1:01
  • Besides what @vonbrand said, you could get something like "Too many connections" and not be able to download all files. And it can be a bit slower (for example reusing one HTTP connection compared to creating several HTTP connections) – golimar Feb 6 '14 at 15:10
  • 2
    As long as you keep the number sane, it's not a big deal. E.g., at the time you wrote this, Firefox was using 15 connections per server when not using persistent connections (they've since switched to only trying persistent connections, which are limited to 6 per server). Other browsers use similar numbers. – derobert Jun 24 '15 at 3:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.