How to download multiple files at once from a web site

Question

How can I download multiple files at once from web page. For example I want to download all the plugins at once from this page.

What I did until now is that every time I needed the file url I would use left click on a file and copy link address and then I would use wget and past the address. This is very tiresome job to do.

Answer 1

If you wanted to download all the same file extentions ( same globbing rules ) in a remote directory you can use this:

 wget -r -l1 --no-parent --no-check-certificate  -Aperl-*.rpm https://mirror.cpsc.ucalgary.ca/mirror/fedora/linux/releases/28/Everything/x86_64/os/Packages/p/ 

This downloads all the perl-*.rpm packages from a directory.

Answer 2

Create a links.txt file containing URLs, and then run

while IFS= read -r line; do
    wget "$line"
done <links.txt

This will run wget command for each link in the file.

Answer 3

you can anaylse a bit the returned htms of your target page and play a bit with bash utils. This should work:

for i in $(curl https://sourceforge.net/projects/geoserver/files/GeoServer/2.10.1/extensions/  | grep net.sf.files | awk -F "=" '{print $2}' | jq '.[].full_path' | awk -F '"' '{printf("https://sourceforge.net/projects/geoserver/files/%s\n",$2)}') ; do curl -o $(echo $i | awk -F '/' '{print $NF}') -L ${i} ; done

Answer 4

As an alternative you can use wget (which is not quite as portable):

curl -sS https://sourceforge.net/projects/geoserver/files/GeoServer/2.10.1/extensions/ | sed -rn 's_^.*<a href="(https://sourceforge.net/projects/geoserver/files/.*/download)"$_\1_p' | wget -i - --trust-server-names

• The curl simply retrieves the html page containing the list of files.
• The sed command finds the urls and strips them out.
• The wget then downloads the list of files.

It might be easier to understand by putting it into a (somewhat reusable) script called:

#!/bin/bash
if  [[ -z "$1" || -z "$2" || -n "$3" && "$3" != 'test' ]]; then
    echo "Usage:    dlist <htmlsource> <regexp> [test]\n    <htmlsource> can be a local file or url\n   <regex> is the inner part to \"s_^.*${regex}.*\$_\1_p\"\n   [test] is the option to show the files to be download"
else
    regex="s_^.*$2.*\$_\1_p"
    if [[ "$3" == 'test' ]]; then
        echo "Using regex $regex:"
        regex="s_^.*$2.*\$_ \1_p"
        if [[ "$1" == http* ]]; then
            curl -sS "$1" | sed -rn "$regex"
        else
            sed -rn "$regex" "$1"
        fi
    elif [[ "$1" == http* ]]; then
        curl -sS '$1' | sed -rn "$regex" | wget -i - --trust-server-names
    else
        sed -rn "$regex" "$1" | wget -i - --trust-server-names
    fi
fi

which can be called with (assuming it is named dlist):

dlist https://sourceforge.net/projects/geoserver/files/GeoServer/2.10.1/extensions/ '<a href="(https://sourceforge.net/projects/geoserver/files/.*/download)"'

The first option is simply the url of the list of files you want: you can also save the file and specify that on the commandline.

Then if you view source on the file list web page (or the copy you downloaded), scroll through and find the list of links which will give you the url to search for. When crafting the regex, surround the part of the url you want with brackets (starting from just before the http) and replace the part that changes with all the urls with .*

The second option is the simplified regex crafted above.