I need to download all PDF files that are linked from this webpage https://www.in.tum.de/i07/lehre/ss22/theo/.

Everything I have tried so far either does not get the files or downloads the entire website recursively.

I however am only interested in the PDFs linked to directly from this PAGE (not website).


3 Answers 3


You can use wget's --no-parent (-np) and --level=depth (-l) options to control how much of the site will be mirrored by the -r option. The --no-host-directories (-nH) and --no-directories (-nd) options will also prevent wget from duplicating the remote directory structure. (Double hyphen precedes options when they are written in full, , while single hyphen precedes options when they are written in short form --no-parent is -np)

e.g. something like this:

wget -r -l 1 -nH -nd -np --ignore-case -A '*.pdf' https://www.in.tum.de/i07/lehre/ss22/theo/

By default, that will save the .pdf files in the current directory. You can use the -P option to specify a different output dir.

wget is very flexible and has a lot of options - so many that the man page can be overwhelming when you first read it, but it is definitely worth putting in some effort to read and experiment with.


You can extract the list of PDF files from the web page using wget or curl to download it, and xmlstarlet to parse the resulting HTML/XML:

curl https://www.in.tum.de/i07/lehre/ss22/theo/ |
    xmlstarlet format -H index.html 2>/dev/null |
    xmlstarlet select -t -m '//a[contains(@href,"pdf")]' -v '@href' -n

The first xmlstarlet converts HTML to XML. The second one find all the a elements and extracts each href attribute value that contains pdf.

From there it's straightforward to download each of the extracted links. Pipe the output of the preceding block into a loop

while IFS= read -r url
    file="${url%\?*}"                            # Strip trailing ? parameters
    file="${file##*/}"                           # Strip leading URL path
    printf "Saving %s as %s\n" "$url" "$file"    # Report action
    curl "$url" >"$file"

More broadly, You can use wget to download all PDFs from a webpage by using:

wget -r -l1 -H -t1 -nd -N -np -A.pdf -erobots=off --wait=2 --random-wait --limit-rate=20k [URL]
  • -r: Recursive download.
  • -l1: Only one level deep (i.e., only files directly linked from this page).
  • -H: Span hosts (follow links to other hosts).
  • -t1: Number of retries is 1.
  • -nd: Don't create a directory structure, just download all the files into the current directory.
  • -N: Turn on timestamping.
  • -np: Do not follow links to parent directories.
  • -A.pdf: Accept only files that end with .pdf.
  • -erobots=off: Ignore the robots.txt file (use carefully, respecting site's terms and conditions).
  • --wait=2: Wait 2 seconds between each retrieval.
  • --random-wait: Wait from 0.5 to 1.5 * --wait seconds between retrievals.
  • --limit-rate=20k: Limit the download rate to 20 kilobytes per second.

This parameters will avoid the "429: Too Many Requests" error.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .