Is possible with curl or wget to automate a download url?

Question

I usually build packages for my Slackware, with SlackBuilds. I download the source with wget, and I use the variables.

With a link like this, no problem when the version change, instead of 16.0.1, I put 17.0.1 and it download automatically, then I build...etc..

VERSION=16.0.1
https://download.java.net/java/GA/jdk${VERSION}/GPL/openjdk-${VERSION}_linux-x64_bin.tar.gz

The problem is when in the url appear some random number which change for every release! In this example are 7147401fd7354114ac51ef3e1328291f and 9

VERSION=16.0.1
https://download.java.net/java/GA/jdk${VERSION}/7147401fd7354114ac51ef3e1328291f/9/GPL/openjdk-${VERSION}_linux-x64_bin.tar.gz

I can use variable substitution, but is a little bad, someone know a more elegant method to get the new "random" number when they change it? With wget or curl.

Answer 1

You can use 2 wgets:

1. wget an HTML page from the website that contains the list of files available for download. Then extract each href=....
2. Filter those hrefs for patterns that match what you are looking for. This might be something like:

https://download.java.net/java/GA/jdk(*)/*/[0-9]/GPL/openjdk-(*)_linux-x64_bin.tar.gz

3. Use capture groups (example: ([0-9]+\.[0-9]+)) to extract the version from each URL

4. Select which version you want to download. This might be the highest version, or it could be the version matching your variable ${VERSION}

5. wget that url

---

This is very similar to the uscan tool which is used by debian. uscan will use a file called debian/watch to do exactly what I mentioned above, except it will usually either identify the latest version of a package (and inform developers that one of their packages has an update available), or download it immediately so the developer can import it into debian.

uscan does a bit more than just that, including renaming the downloaded archive to match the name of the package as it appears in debian. That depends on some other proximate files existing, so uscan may not work out-of-the-box for you. It at least suggests that the design of downloading the index, parsing the hrefs, selecting the best one, and wget-ing that works pretty well. uscan is a perl script which isn't too complicated (~4000 sloc with support for lots of stuff you can purge) so you may be able adapt it for your needs.