I want to download all the pdf files in this website:


The command I run is:

wget -r -A "*.pdf" "http://www.math.utsc.utoronto.ca/b41/"

However, if you go to the website, there is an "assignments" tab, which has assignments 1-10, it does not download those for some reason? It's still on the same domain, http://www.math.utsc.utoronto.ca/b41/a*.pdf, where 1 <= * <= 10, but wget does not download this?

  • a cursory peep at the source under "math.utsc.utoronto.ca/b41/"; shows all those "a*.pdf" as value attributes of html option tags. i don't know whether wget can fetch those. but trivially you could write a bash one-liner with curl to fetch them all. something akin to: for Q in $(seq 1 23);do curl "math.utsc.utoronto.ca/b41/a${Q}";done (adding http:// in front of that url) Dec 8, 2017 at 21:27

2 Answers 2


From manpage of wget:

With HTTP URLs, Wget retrieves and parses the HTML or CSS from the given URL, retrieving the files the document refers to, through markup like href or src, or CSS URI values specified using the ‘url()’ functional notation. If the freshly downloaded file is also of type text/html, application/xhtml+xml, or text/css, it will be parsed and followed further.

In other words, if the file location is not explicitly written as href or src or something alike in the in the source of the page, but instead generated by some JavaScript snippets or are returned (by php, for example) after submitting some requests to the server, there are not much wget can do with it even if it is stored on the same domain.


One factor to take into account is that wget treats any difference in the domain name as a separate domain and will not deviate from the supplied domain by default.

Ie. if you ask it to download https://example.com/somefile.html, then it will not follow any internal links it finds in the form <a href="https://www.example.com...">.

In this case it would be necessary to supply --span-hosts --domains=example.com

Note that it is does not seem to necessary to explicitly supply www.example.com to the domains argument as in this case it seems to match automatically.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .