Is there parallel wget? Something like fping but only for downloading?

Question

I've found only puf (Parallel URL fetcher) but could to get it work with reading urls from file and something like

 puf < urls.txt

does not work either.

The operating system installed on the server is ubuntu.

Answer 1

You can implement that using Python and the pycurl library. The pycurl library has the "multi" interface that implements its own even loop that enables multiple simultaneous connections.

However the interface is rather C-like and therefore a bit cumbersome as compared to other, more "Pythonic", code.

I wrote a wrapper for it that builds a more complete browser-like client on top of it. You can use that as an example. See the pycopia.WWW.client module. The HTTPConnectionManager wraps the multi interface.

Answer 2

Using GNU Parallel,

$ parallel -j ${jobs} wget < urls.txt

or xargs from GNU Findutils,

$ xargs -P ${jobs} wget < urls.txt

where ${jobs} is the maximum number of wget you want to allow to run concurrently. Without -j/-P, parallel will run as many jobs at a time as CPU cores (which doesn't necessarily make sense for wget bound by network IO), and xargs will run one at a time.

One nice feature that parallel has over xargs is keeping the output of the concurrently-running jobs separated, but if you don't care about that, xargs is more likely to be pre-installed.

Answer 3

Part of GNU Parallel's man page contains an example of a parallel recursive wget.

https://www.gnu.org/software/parallel/man.html#example__breadth_first_parallel_web_crawler_mirrorer

Otherwise ephemient's answer seems obvious.

Answer 4

One solution would be something like the following:

#!/bin/sh
while IFS= read -r uri; do
    wget -q -- "${uri}" &
done

Then run it as ./puf < urls.txt.

Answer 5

aria2 does this.

http://sourceforge.net/apps/trac/aria2/wiki/UsageExample#Downloadfileslistedinafileconcurrently

Answer 6

The victims of your paralell download won't be amused: they expect one connection to serve each client, setting up several connections means less clients overall. (I.e., this is considered rude behaviour).