0

How can I download multiple files at once from web page. For example I want to download all the plugins at once from this page.

What I did until now is that every time I needed the file url I would use left click on a file and copy link address and then I would use wget and past the address. This is very tiresome job to do.

  • 1
    wget has builtin functionality to follow/expand selectable web links that handles many cases like this, but I haven't tested this one specifically. See the man page starting at Recursive Retrieval Options or the info at the equivalent topics under Invoking and addtional discussion at Following Links – dave_thompson_085 Jan 15 '17 at 14:01
1

you can anaylse a bit the returned htms of your target page and play a bit with bash utils. This should work:

for i in $(curl https://sourceforge.net/projects/geoserver/files/GeoServer/2.10.1/extensions/  | grep net.sf.files | awk -F "=" '{print $2}' | jq '.[].full_path' | awk -F '"' '{printf("https://sourceforge.net/projects/geoserver/files/%s\n",$2)}') ; do curl -o $(echo $i | awk -F '/' '{print $NF}') -L ${i} ; done
|improve this answer|||||
1

As an alternative you can use wget (which is not quite as portable):

curl -sS https://sourceforge.net/projects/geoserver/files/GeoServer/2.10.1/extensions/ | sed -rn 's_^.*<a href="(https://sourceforge.net/projects/geoserver/files/.*/download)"$_\1_p' | wget -i - --trust-server-names
  • The curl simply retrieves the html page containing the list of files.
  • The sed command finds the urls and strips them out.
  • The wget then downloads the list of files.

It might be easier to understand by putting it into a (somewhat reusable) script called:

#!/bin/bash
if  [[ -z "$1" || -z "$2" || -n "$3" && "$3" != 'test' ]]; then
    echo "Usage:    dlist <htmlsource> <regexp> [test]\n    <htmlsource> can be a local file or url\n   <regex> is the inner part to \"s_^.*${regex}.*\$_\1_p\"\n   [test] is the option to show the files to be download"
else
    regex="s_^.*$2.*\$_\1_p"
    if [[ "$3" == 'test' ]]; then
        echo "Using regex $regex:"
        regex="s_^.*$2.*\$_ \1_p"
        if [[ "$1" == http* ]]; then
            curl -sS "$1" | sed -rn "$regex"
        else
            sed -rn "$regex" "$1"
        fi
    elif [[ "$1" == http* ]]; then
        curl -sS '$1' | sed -rn "$regex" | wget -i - --trust-server-names
    else
        sed -rn "$regex" "$1" | wget -i - --trust-server-names
    fi
fi

which can be called with (assuming it is named dlist):

dlist https://sourceforge.net/projects/geoserver/files/GeoServer/2.10.1/extensions/ '<a href="(https://sourceforge.net/projects/geoserver/files/.*/download)"'

The first option is simply the url of the list of files you want: you can also save the file and specify that on the commandline.

Then if you view source on the file list web page (or the copy you downloaded), scroll through and find the list of links which will give you the url to search for. When crafting the regex, surround the part of the url you want with brackets (starting from just before the http) and replace the part that changes with all the urls with .*

The second option is the simplified regex crafted above.

|improve this answer|||||
1

If you wanted to download all the same file extentions ( same globbing rules ) in a remote directory you can use this:

 wget -r -l1 --no-parent --no-check-certificate  -Aperl-*.rpm https://mirror.cpsc.ucalgary.ca/mirror/fedora/linux/releases/28/Everything/x86_64/os/Packages/p/ 

This downloads all the perl-*.rpm packages from a directory.

|improve this answer|||||

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.