I want to download all the pdf files in this website:


The command I run is:

wget -r -A "*.pdf" "http://www.math.utsc.utoronto.ca/b41/"

However, if you go to the website, there is an "assignments" tab, which has assignments 1-10, it does not download those for some reason? It's still on the same domain, http://www.math.utsc.utoronto.ca/b41/a*.pdf, where 1 <= * <= 10, but wget does not download this?

  • a cursory peep at the source under "math.utsc.utoronto.ca/b41/"; shows all those "a*.pdf" as value attributes of html option tags. i don't know whether wget can fetch those. but trivially you could write a bash one-liner with curl to fetch them all. something akin to: for Q in $(seq 1 23);do curl "math.utsc.utoronto.ca/b41/a${Q}";done (adding http:// in front of that url) – Theophrastus Dec 8 '17 at 21:27

From manpage of wget:

With HTTP URLs, Wget retrieves and parses the HTML or CSS from the given URL, retrieving the files the document refers to, through markup like href or src, or CSS URI values specified using the ‘url()’ functional notation. If the freshly downloaded file is also of type text/html, application/xhtml+xml, or text/css, it will be parsed and followed further.

In other words, if the file location is not explicitly written as href or src or something alike in the in the source of the page, but instead generated by some JavaScript snippets or are returned (by php, for example) after submitting some requests to the server, there are not much wget can do with it even if it is stored on the same domain.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.