Download arbitrary number of files using wget from a self-redirecting directory

Question

I need to download a very large number of pdf files (would be tedious if not impossible to collect and collate all the links) from my university website using wget, with links like this:

http://example.edu/path_to_directory/directory/foobar.pdf

I am using the following command:

wget -mk --no-check-certificate --load-cookies cookies.txt  http://example.edu/arbitrary_link_to_directory/directory/

Unfortunately the actual directories which contain the pdfs each redirect to an error page, and all I get are a bunch of html and css files.

http://example.edu/path_to_directory/directory/

would in a browser redirect one to

http://example.edu/path_to_directory/error/error.html

Would it be possible to download the all files in question using wget at all?

Answer 1

When I want to download a bunch of files from a web site that doesn't make it easy (or actively makes it difficult), I generate a list of the files' URLs with something like this:

lynx -dump -listonly $URL | awk '/\.pdf/ {print $2}' > /tmp/pdflist.txt

I then usually edit the file with vi to delete entries I don't want to download, and finally fetch them all with wget:

wget --input-file=/tmp/pdflist.txt

That works well for simple tasks, and often works well enough for moderately difficult tasks...but for difficult tasks involving parsing of convoluted html and following links (and maybe using cookies), I'll write a perl web-bot using the libwww-perl module (aka LWP).

Answer 2

On your university website, there is a page providing links to all PDFs? Or how do you know the list of PDF file names?

In that case, you will need to write a script called scraper to programmatically get links from that page and then automatically loop through that list and download them with wget or another tool. I would suggest PHP or Python to write the scraper. You will easy manage the eventual pagination on the origin website. If using PHP, you can simply download files with

file_put_contents('destination_filename', file_get_contents('source_url');