1

I need to download a very large number of pdf files (would be tedious if not impossible to collect and collate all the links) from my university website using wget, with links like this:

http://example.edu/path_to_directory/directory/foobar.pdf

I am using the following command:

wget -mk --no-check-certificate --load-cookies cookies.txt  http://example.edu/arbitrary_link_to_directory/directory/

Unfortunately the actual directories which contain the pdfs each redirect to an error page, and all I get are a bunch of html and css files.

http://example.edu/path_to_directory/directory/

would in a browser redirect one to

http://example.edu/path_to_directory/error/error.html

Would it be possible to download the all files in question using wget at all?

1
  • 1
    If your site is spidered by google you could do a google search forsite:example.edu/dir/ filetype:pdf and get a list of urls that might work better. – meuh Nov 5 '15 at 17:17
2

When I want to download a bunch of files from a web site that doesn't make it easy (or actively makes it difficult), I generate a list of the files' URLs with something like this:

lynx -dump -listonly $URL | awk '/\.pdf/ {print $2}' > /tmp/pdflist.txt

I then usually edit the file with vi to delete entries I don't want to download, and finally fetch them all with wget:

wget --input-file=/tmp/pdflist.txt

That works well for simple tasks, and often works well enough for moderately difficult tasks...but for difficult tasks involving parsing of convoluted html and following links (and maybe using cookies), I'll write a perl web-bot using the libwww-perl module (aka LWP).

2
  • Thanks. Unfortunately, most of the links on that website are all over the place inside numerous pages, and inside a frame (!!!). Is there a way to scrape links here? – fernsehen.f1 Nov 13 '15 at 22:58
  • Write a web bot, e.g. in perl using LWP, as i mentioned. – cas Nov 13 '15 at 23:02
0

On your university website, there is a page providing links to all PDFs? Or how do you know the list of PDF file names?

In that case, you will need to write a script called scraper to programmatically get links from that page and then automatically loop through that list and download them with wget or another tool. I would suggest PHP or Python to write the scraper. You will easy manage the eventual pagination on the origin website. If using PHP, you can simply download files with

file_put_contents('destination_filename', file_get_contents('source_url');

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.