3

This may/may not be possible using bash happy for a suggestion in Python if there is one.

But I would like to download all the PDFs/ePubs from the following link

https://mises.org/library/books

You can select the PDFs that they have available by doing the following by selecting PDF and then ALL. You will see a list of the documents they have in PDF are listed.

The problem is that the URL does not seem to change as you go up through the list , also the same link is there if you chose EPUB and then choose ALL.

So if I was to use wget curl there does not seem to be a way to distinguish PDF from EPUB.

I am then going to convert them to PDF/EPubs using Caliber and upload to my Kindle account so then I can have a stack of them to read while on the tube.

Thanks in advance..

  • If you have done a google search with the keywords: wget download all pdf documents on a web site, you could have found this page easily: question-defense.com/2010/12/22/… – MelBurslan Jan 20 '16 at 15:16
  • @MelBurslan I converted your answer to a comment since it i) was a link only answer and ii) it doesn't actually help the OP. If you had read the page you linked to and tried it on the OP's target URL, you would have seen that. – terdon Jan 20 '16 at 16:37
  • 1
    Are any of the books listed there actually available? The link you give has no PDFs or ePUBs, only links to where you can buy them. How are you expecting to download them? – terdon Jan 20 '16 at 16:40
  • @terdon I have updated my question so that it answers you points. Also just to be sure I went to link (mises.org/library/books )and selected PDF and then selected ALL and then downloaded the first 10 PDFs. – user1380599 Jan 20 '16 at 19:17
3

(Started out as a few hints, ended up as a rather huge blob of an answer. (again :]))

I can not answer for if there are any policy on the site against batch downloading, but as a start.

Though this is very site specific the same procedure should be applicable to similar sites thus the answer might be useful for others. More like an example.


On that kind of page I would have looked closer at the site, and not simply tried to crawle it as a whole. For one there are both free and paid contents. Secondly; merely downloading all in a huge pile might become somewhat messy, especially considering how many documents there are.

Isolating the index pages

  1. The page show the same URL no matter what section you are in. But either by looking at source or simply hoovering one of the sections you'll see it has at least two likely interesting targets:

    PDF : https://mises.org/library/books?book_type=539
    EPUB: https://mises.org/library/books?book_type=537
    
  2. The result is split by 10 and 10 hits. For example: PDF show Displaying 1 - 10 of 535. Hoovering the numbers at bottom we locate the next interesting piece of information: &page=1, &page=2 etc.

As there are 535 hits we have to download 535 / 10 = 54 pages. As page=0 the actual last page would be page=53.

This gives us a hint on what to download to get started. PDF:

https://mises.org/library/books?book_type=539&page=0
https://mises.org/library/books?book_type=539&page=1
...
https://mises.org/library/books?book_type=539&page=53

As free PDF's seems to include free EPUB's there is likely no need to do the same for the EPUB tab.

In this case one might end up with doing something like:

mkdir mises.org
cd mises.org

Either execute in terminal or add to a script file:

for i in {0..53}
do
    curl -o ./"$(printf "pdfl-%03d.html" $i)" "https://mises.org/library/books?book_type=539&page=$i"
done

Already we have a nice ordered structure to work with.

Extracting relevant data

After this is done the next step is to look closer at one of the sub-pages. For example pdfl-000.html.

After a quick examination we find that each book is enclosed within a:

  <div class="panel-body">
     ... book data ...
  </div>

Now all depending on what you'd like to extract the path from here on is either to go for file-name + file-url, or also include meta data and other sugar such as author, topic, images etc. For the latter you would use a HTML parser in something like Python, Perl or the like. For the former this could also be used - but one could also go for simpler text processing tools like sed, grep etc.

In this case, (for simplicity), I go for the former.

Now it is often stated that on should never ever parse HTML with regexp etc., but in this case we are not parsing the document but merely extracting relevant data. Thus, especially since it is not a random possibly malicious site, nor a hack-splash of crazy markup - but concise and well formatted - the use of sed etc. is OK (IMHO).

Looking further into pdfl-000.html we find that title is enclosed in <h2 class="element-invisible"> ending with .pdf. The link for the document <a, has type="application/pdf.

With this as base we can quickly extract all titles and targets with for example sed. As the result of sed gives us title and URI on alternating lines we can pipe the result to paste - -.

As numbering the items would be nice we further pipe it to awk to add some altered document name. This also likely saves us if two document should have the same title.

In this crude example we assume no names has tab, newline or " as part of it.

sed -n -e 's#.*<h2 class="element-invisible"><a [^>]*>\([^<]*\.pdf\)</a>.*</h2>#\1#p' \
    -e 's#.*<a[^>]* href="\([^"]*\)"[^<]*type="application/pdf;.*#"https://mises.org\1"#p' \
    pdfl-* | \
    paste - - | \
    awk -F'\t' '{ printf "curl -o \"%03d-%s\" %s\n",NR,$1,$2 }' > curl.txt

Download

Then it is simply to do:

mkdir pdf
cd pdf
. ../curl.txt

and wait for it to download. Ending up with something like:

001-A Spontaneous Order.pdf
002-Science Technology and Government.pdf
003-Science Technology and Government—Chinese.pdf
004-The Next Generation of Austrian Economics Essays in Honor of Joseph T. Salerno.pdf
005-A Short History of Man — Progress and Decline.pdf
...

Side note

  • Keep the pdfl-NNN.html files if you ever want to extract more information from the files.

  • A quick sum of all PDF's using the lenghth=NNN value of the URL's show a total of 7401933553 bytes or 6.89 GiB to be downloaded :P

  • Amazing thanks for the response it worked perfectly !! – user1380599 Jan 21 '16 at 17:08
  • I do have one follow up question. – user1380599 Feb 2 '16 at 19:35
  • @user1380599: If might be best as its own question ... What is it? – Runium Feb 3 '16 at 18:44
  • sorry about that I thought I had filled in the section . – user1380599 Feb 11 '16 at 16:32
  • @user1380599: ah. OK. Anyhow: If questions arises that are directly related to question/answer ask away. Do not ask to ask :P Hope it got solved. – Runium Feb 11 '16 at 21:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.