3

I wanted to download all graphic files from our organisation's graphics repository web page. They are Illustrator (.ai) format and Corel Draw (.cdr) format.

They are directly hyperlinked (i.e. <a href="http://server/path-to-file.ai">...</a>.

7

wget includes features to support this directly:

wget -r -A "*.ai,*.cdr" 'address-of-page-with-hyperlinks'

-r enables recursive mode so it will download more than the given URL, and -A limits the files it will download and keep in the end.

  • 1
    Nice, I knew about wget's spidering capabilities but never knew it had such a nice filtering option! – Mark K Cowan Mar 9 '15 at 15:26
0
  1. Read the page with curl

  2. Filter for hyperlinks with grep. Note that this exploits the fact that they are all formatted consistently (as <a href="..."). An HTML/XML parser really should be used for anything even slightly more complicated.

  3. Cut the hyperlink address out with grep -o, filtering with grep again to extract only links who's target has the desired extension(s) and protocol(s).

  4. Download those resulting files with wget (or curl could also be used if output was specified). Also, we echo the URL to the the console.

Note that our filtering in the script is not in the exact same order as above, for convenience.

Bash:

#!/bin/bash
while read url; do
    # Echo URL to terminal
    echo "${url}"
    # Download files
    wget -q "${url}"
done < <(
    # Get page
    curl -s 'address-of-page-with-hyperlinks' | \
    # Filter hyperlinks
    grep -Po 'href="http://[^"]+\.(cdr|ai)"' | \
    grep -Po 'http://[^"]+'
)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.