3

I'm trying to download this published journal article using cURL. It's the main page of an open access, so there should be not problems for anyone to see/download the article. I then extract the pdfurl, which keeps changing.

Then I try to download the pdf:

curl -L -o test.pdf "http://www.sciencedirect.com/science/article/pii/S0378426612000817/pdfft?md5=6a85f34def09dd5cfb1d1b8feded0d51&pid=1-s2.0-S0378426612000817-main.pdf"

but all the time it redirects me to the main page, which is then downloaded as a html page called "test.pdf".

6
  • Have you tried using wget instead of curl?
    – Nik-Nak
    Apr 24, 2015 at 8:46
  • It seems that this is a Science Direct download link, that is not working for some people. What is the page where you got the PDF URL from? Are you accessing from some organization or through some proxy service that gives you additional privileges at Science Direct?
    – njsg
    Apr 24, 2015 at 9:05
  • I just tried this here... can you try with curl -L --referer ";auto" -o test.pdf URL-for-direct-download? It seems that the URL involves a redirect, and it also needs the referer header to be set correctly.
    – njsg
    Apr 24, 2015 at 9:09
  • Why should that help? Apr 24, 2015 at 9:41
  • wget supports retrieval through proxies, in my mind it is the better tool for the job
    – Nik-Nak
    Apr 24, 2015 at 10:19

2 Answers 2

4

curl seems to handle redirects differently from wget by default. The direct download URL will involve some redirects and it also requires the HTTP referer header to be set correctly after the first redirect (otherwise, you will get a HTML page).

First, you need to enable location redirects in curl with -L, and then enable curl's automatic handling of the referer header with --referer ";auto", that is,

curl -L --referer ";auto" -o test.pdf URL-for-direct-download
2
  • I recommend cliget addon for Firefox that takes care of all of this automatically Apr 24, 2015 at 9:33
  • Seems like a nice addon, @ArkadiuszDrabczyk. However, I would like to download the files given a list of DOIs from a remote server.
    – MERose
    Apr 24, 2015 at 11:59
0

I managed to get this fine

wget -O main.pdf "http://ac.els-cdn.com/S0378426612000817/1-s2.0-S0378426612000817-main.pdf?_tid=5eb365f4-ea5e-11e4-b707-00000aacb360&acdnat=1429865356_af9bf791080efa05beb6e88a2c82b41b"

2
  • Did you download a pdf? Because I get redirected to the main page.
    – MERose
    Apr 24, 2015 at 8:54
  • Nope I definitely got a PDF. However when I tried it again I get a 404, the URL keeps changing. Saving to: ‘main.pdf’ main.pdf 100%[==================================================================>] 391.42K --.-KB/s in 0.07s 2015-04-24 10:01:20 (5.79 MB/s) - ‘main.pdf’ saved [400817/400817] $ file main.pdf main.pdf: PDF document, version 1.7
    – Aditya K
    Apr 24, 2015 at 9:03

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .