3

I'm trying to download this published journal article using cURL. It's the main page of an open access, so there should be not problems for anyone to see/download the article. I then extract the pdfurl, which keeps changing.

Then I try to download the pdf:

curl -L -o test.pdf "http://www.sciencedirect.com/science/article/pii/S0378426612000817/pdfft?md5=6a85f34def09dd5cfb1d1b8feded0d51&pid=1-s2.0-S0378426612000817-main.pdf"

but all the time it redirects me to the main page, which is then downloaded as a html page called "test.pdf".

  • Have you tried using wget instead of curl? – Nik-Nak Apr 24 '15 at 8:46
  • It seems that this is a Science Direct download link, that is not working for some people. What is the page where you got the PDF URL from? Are you accessing from some organization or through some proxy service that gives you additional privileges at Science Direct? – njsg Apr 24 '15 at 9:05
  • I just tried this here... can you try with curl -L --referer ";auto" -o test.pdf URL-for-direct-download? It seems that the URL involves a redirect, and it also needs the referer header to be set correctly. – njsg Apr 24 '15 at 9:09
  • Why should that help? – roaima Apr 24 '15 at 9:41
  • wget supports retrieval through proxies, in my mind it is the better tool for the job – Nik-Nak Apr 24 '15 at 10:19
4

curl seems to handle redirects differently from wget by default. The direct download URL will involve some redirects and it also requires the HTTP referer header to be set correctly after the first redirect (otherwise, you will get a HTML page).

First, you need to enable location redirects in curl with -L, and then enable curl's automatic handling of the referer header with --referer ";auto", that is,

curl -L --referer ";auto" -o test.pdf URL-for-direct-download
  • I recommend cliget addon for Firefox that takes care of all of this automatically – Arkadiusz Drabczyk Apr 24 '15 at 9:33
  • Seems like a nice addon, @ArkadiuszDrabczyk. However, I would like to download the files given a list of DOIs from a remote server. – MERose Apr 24 '15 at 11:59
0

I managed to get this fine

wget -O main.pdf "http://ac.els-cdn.com/S0378426612000817/1-s2.0-S0378426612000817-main.pdf?_tid=5eb365f4-ea5e-11e4-b707-00000aacb360&acdnat=1429865356_af9bf791080efa05beb6e88a2c82b41b"

  • Did you download a pdf? Because I get redirected to the main page. – MERose Apr 24 '15 at 8:54
  • Nope I definitely got a PDF. However when I tried it again I get a 404, the URL keeps changing. Saving to: ‘main.pdf’ main.pdf 100%[==================================================================>] 391.42K --.-KB/s in 0.07s 2015-04-24 10:01:20 (5.79 MB/s) - ‘main.pdf’ saved [400817/400817] $ file main.pdf main.pdf: PDF document, version 1.7 – Aditya K Apr 24 '15 at 9:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.