Get the file list in an HTML directory listing

Question

I can get all files on the bash patches site by downloading them in a sequence:

SEQ=$(seq -f "%03g" 1 30)
for i in $SEQ; do 
  wget http://ftp.gnu.org/gnu/bash/bash-4.3-patches/bash43-$i;
done

But then I would have to know the maximum number.

Is there a possibility to just get the listing and extract all patchfiles for downloading?

Answer 1

You could use wget with recursive downloading:

wget -nc -nd -nH -np -r -R '*.*'  http://ftp.gnu.org/gnu/bash/bash-4.3-patches/

Explanation:

• -nc: no-clobber (don't overwrite existing files), probably not necessary.
• -nd: Don't create hierarchy of directories.
• -nH: Don't create directory based on hostname. Or you'd find everything downloaded to a directory called ftp.gnu.org.
• -np: Never ascend to the parent directory.
• -r: Download recursively.
• -R '*.*': Reject everything with a . in its filename (skips things like index.html and so on). An accept list may also be used. The file is downloaded, but discarded.

Answer 2

for i in $(seq -f "%03g" 1 999); do 
  wget http://ftp.gnu.org/gnu/bash/bash-4.3-patches/bash43-$i
  if [[ $? -ne "0" ]]; then
    MAX=$(expr $i - 1)
    break;
  fi
done
echo $MAX files downloaded