How can I download multiple files at once from web page. For example I want to download all the plugins at once from this page.

What I did until now is that every time I needed the file url I would use left click on a file and copy link address and then I would use wget and past the address. This is very tiresome job to do.

  • 1
    wget has builtin functionality to follow/expand selectable web links that handles many cases like this, but I haven't tested this one specifically. See the man page starting at Recursive Retrieval Options or the info at the equivalent topics under Invoking and addtional discussion at Following Links Commented Jan 15, 2017 at 14:01

4 Answers 4


If you wanted to download all the same file extentions ( same globbing rules ) in a remote directory you can use this:

 wget -r -l1 --no-parent --no-check-certificate  -Aperl-*.rpm https://mirror.cpsc.ucalgary.ca/mirror/fedora/linux/releases/28/Everything/x86_64/os/Packages/p/ 

This downloads all the perl-*.rpm packages from a directory.


Create a links.txt file containing URLs, and then run

while IFS= read -r line; do
    wget "$line"
done <links.txt

This will run wget command for each link in the file.


you can anaylse a bit the returned htms of your target page and play a bit with bash utils. This should work:

for i in $(curl https://sourceforge.net/projects/geoserver/files/GeoServer/2.10.1/extensions/  | grep net.sf.files | awk -F "=" '{print $2}' | jq '.[].full_path' | awk -F '"' '{printf("https://sourceforge.net/projects/geoserver/files/%s\n",$2)}') ; do curl -o $(echo $i | awk -F '/' '{print $NF}') -L ${i} ; done

As an alternative you can use wget (which is not quite as portable):

curl -sS https://sourceforge.net/projects/geoserver/files/GeoServer/2.10.1/extensions/ | sed -rn 's_^.*<a href="(https://sourceforge.net/projects/geoserver/files/.*/download)"$_\1_p' | wget -i - --trust-server-names
  • The curl simply retrieves the html page containing the list of files.
  • The sed command finds the urls and strips them out.
  • The wget then downloads the list of files.

It might be easier to understand by putting it into a (somewhat reusable) script called:

if  [[ -z "$1" || -z "$2" || -n "$3" && "$3" != 'test' ]]; then
    echo "Usage:    dlist <htmlsource> <regexp> [test]\n    <htmlsource> can be a local file or url\n   <regex> is the inner part to \"s_^.*${regex}.*\$_\1_p\"\n   [test] is the option to show the files to be download"
    if [[ "$3" == 'test' ]]; then
        echo "Using regex $regex:"
        regex="s_^.*$2.*\$_ \1_p"
        if [[ "$1" == http* ]]; then
            curl -sS "$1" | sed -rn "$regex"
            sed -rn "$regex" "$1"
    elif [[ "$1" == http* ]]; then
        curl -sS '$1' | sed -rn "$regex" | wget -i - --trust-server-names
        sed -rn "$regex" "$1" | wget -i - --trust-server-names

which can be called with (assuming it is named dlist):

dlist https://sourceforge.net/projects/geoserver/files/GeoServer/2.10.1/extensions/ '<a href="(https://sourceforge.net/projects/geoserver/files/.*/download)"'

The first option is simply the url of the list of files you want: you can also save the file and specify that on the commandline.

Then if you view source on the file list web page (or the copy you downloaded), scroll through and find the list of links which will give you the url to search for. When crafting the regex, surround the part of the url you want with brackets (starting from just before the http) and replace the part that changes with all the urls with .*

The second option is the simplified regex crafted above.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .