3

So wget has an ability to recursively download files, however it does it one file at a time.

I would like to pass in a directory URL, and for each URL it encounters in the recursion for it to spawn off a downloading process.

One way I was thinking to do this is to somehow use wget to print out the URLs it encounters, and then feeding those URLs into separate instances of wget (via wget URL_1 &, wget URL_2 & etc).

Any ideas?

1
  • 2
    I'm not sure if wget is the right tool for the job here. Are you totally dead-set on using wget? In the year 2022, there are probably better tools for getting a list of links from an arbitrary URL, e.g., xmlstarlet, xidel, or pup. Sep 20, 2022 at 23:44

4 Answers 4

1

I've been ruminating on this, and I'm not convinced wget is the best tool for the job here.

Here is how I would do this in the year 2022, using a tool like pup that is specifically designed to parse HTML (in pup's case, with CSS selectors):

wget -q -O- https://ubuntu.com/download/alternative-downloads \
  | pup 'a[href$=".torrent"] attr{href}' \
  | aria2c -d ~/Downloads -i -

See also

  • xidel

    • the -e / --extract option uses XPath selectors by default; supports CSS selectors with --css '<selector>' or --extract 'css("<selector>")'
    • can fetch internet resources directly—a bit slower than curl on my machine, though
    • very tolerant parser; almost never seen it complain, even for malformed HTML
    • examples:
      xidel https://www.videlibri.de/xidel.html \
            -e '//a[ends-with(@href,"/download")]/@href'
      
      # faster, for some reason; don't forget the '-' (read from stdin)!
      curl -q https://www.videlibri.de/xidel.html \
        | xidel -e '//a[ends-with(@href,"/download")]/@href' -
      
      # same as above, using CSS selectors + XPath for the attribute
      curl -q https://www.videlibri.de/xidel.html \
        | xidel -e 'css("a[href$=/download]")/@href' -
      
  • xmlstarlet

    • uses XPath selectors
    • must have well-formed XML/XHTML as input
      • piping through xmlstarlet fo -H -R (reformat, expect input as HTML, try to Recover after errors) should fix for most web sites
    • example:
      # NB: my version of xmlstarlet doesn't support XPath 'ends-with'
      curl -s https://ubuntu.com/download/alternative-downloads \
        | xmlstarlet fo -H -R 2>/dev/null \
        | xmlstarlet sel -t -v '//a[contains(@href, ".torrent")]/@href' -n
      
  • aria2

1

Use grep with it:

wget url -rqO - | grep -oE '[a-z]+://[^[:space:]"]+'
0

One way to solve this problem is to collect all the links in a plain text file and do the following:-

while read line; do
    echo "Downloading ${line}"
    wget $line &        
done < $1

Save this file as script.sh and make it executable and run it as

$ ./script.sh

A better way to permanently solve this problem would be to rewrite wget to be inherently parallel.

0

In my case, it worked by chaining two wgets:

$ wget -O - -o /dev/null http://site.to.get.urls.from | grep -o 'some_regex' | wget -i -

The first wget will produce a list of URLs with files, which is used as input "file" for the second wget.

Note that this is a simplified version to show you the technique behind it. The pipe between the grep and the second wget command might in fact be combined with way more piped commands like sed, awk or cut (especially if the HTML source code is a little more complex than usual and hence parsing it is more difficult).

This reads: Always make sure you call the inner wget standalone first to verify on stdout that your regular expression(s) is (are) working correctly.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .