HTTP Downloader that supports directory index? So that I could use globbing?

Question

Is there an http downloader that supports globbing?

i.e. I would like to fetch the latest update of a package, abc-XXX.rpm, but the XXX (version number) is unknown to me. Is it possible to do a globbing on directory index of the remote server?

Answer 1

lftp sounds like what you need. (despite the name, it supports many protocols including HTTP and extracting file lists from directory indexes).

Answer 2

If the server is an FTP server, wget will expand wildcards:

wget --no-verbose 'ftp://ftp.gnu.org/gnu/tar/*1.26*'
2012-10-02 12:02:17 URL: ftp://ftp.gnu.org/gnu/tar/*1.26* [11217] -> ".listing" [1]
2012-10-02 12:02:23 URL: ftp://ftp.gnu.org/gnu/tar/tar-1.26.cpio.gz [3421172] -> "tar-1.26.cpio.gz" [1]
2012-10-02 12:02:23 URL: ftp://ftp.gnu.org/gnu/tar/tar-1.26.cpio.gz.sig [189] -> "tar-1.26.cpio.gz.sig" [1]
2012-10-02 12:02:31 URL: ftp://ftp.gnu.org/gnu/tar/tar-1.26.shar.gz [5246246] -> "tar-1.26.shar.gz" [1]
2012-10-02 12:02:31 URL: ftp://ftp.gnu.org/gnu/tar/tar-1.26.shar.gz.sig [189] -> "tar-1.26.shar.gz.sig" [1]
2012-10-02 12:02:35 URL: ftp://ftp.gnu.org/gnu/tar/tar-1.26.tar.bz2 [2339773] -> "tar-1.26.tar.bz2" [1]
2012-10-02 12:02:35 URL: ftp://ftp.gnu.org/gnu/tar/tar-1.26.tar.bz2.sig [189] -> "tar-1.26.tar.bz2.sig" [1]
2012-10-02 12:02:41 URL: ftp://ftp.gnu.org/gnu/tar/tar-1.26.tar.gz [3435280] -> "tar-1.26.tar.gz" [1]
2012-10-02 12:02:41 URL: ftp://ftp.gnu.org/gnu/tar/tar-1.26.tar.gz.sig [189] -> "tar-1.26.tar.gz.sig" [1]
2012-10-02 12:02:45 URL: ftp://ftp.gnu.org/gnu/tar/tar-1.26.tar.xz [1783904] -> "tar-1.26.tar.xz" [1]
2012-10-02 12:02:45 URL: ftp://ftp.gnu.org/gnu/tar/tar-1.26.tar.xz.sig [189] -> "tar-1.26.tar.xz.sig" [1]

otherwise, as sch suggests, use lftp:

lftp -c mget 'http://ftp.gnu.org/gnu/tar/*1.26*'