Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

How to download hundereds of .pdf files from http://www.ncbi.nlm.nih.gov/pmc/articles using a loop, for example for the following document ids:

PMC3386155
PMC3625956
PMC3477654
PMC3531051
PMC3114846
PMC3117879
PMC3130560
PMC3531173
PMC3546115
PMC3354575
PMC3771521
share|improve this question
    
Are these open access or do you have to enter your institution's credentials every time? –  terdon Sep 21 '13 at 18:17
1  
related: unix.stackexchange.com/questions/83687/… –  slm Sep 21 '13 at 18:22
    
pubmed Q: unix.stackexchange.com/questions/91696/…. Why is this one different? –  slm Sep 21 '13 at 18:39
    
@sami have you checked that script ? is there any issue ? –  Rahul Patil Sep 22 '13 at 8:55
add comment

1 Answer

Here is Working Tested Script

Using wget

#!/usr/bin/env bash

Link="http://www.ncbi.nlm.nih.gov/pmc/articles/"

ID=(    PMC3386155 PMC3625956 PMC3477654 PMC3531051
        PMC3114846 PMC3117879 PMC3130560 PMC3531173
        PMC3546115 PMC3354575 PMC3771521 )

for f in ${ID[@]};
do
   wget  --user-agent="Mozilla/5.0 (Windows NT 5.2; rv:2.0.1) Gecko/20100101 Firefox/4.0.1" \
         -l1 --no-parent -A.pdf ${Link}${f}/pdf/ -O ${f}.pdf
done

Since Remote site does not allow user agent like wget and curl that's why we have to explicitly specify user agent in wget

Using Curl

ID=( PMC3386155 PMC3625956 PMC3477654 PMC3531051 PMC3114846 PMC3117879 PMC3130560 PMC3531173 PMC3546115 PMC3354575 PMC3771521 )

Link="http://www.ncbi.nlm.nih.gov/pmc/articles/"

Args='-O -J -L -A "Mozilla/5.0 (Windows NT 5.2; rv:2.0.1) Gecko/20100101 Firefox/4.0.1"'

printf "%s\n" ${ID[@]}  | xargs -n1 -I{} echo curl $Args ${Link}'{}'/pdf/ | sh

Some explanation

  • -O Output File
  • -J Output File name from remote-header-name ( curl 7.21.2 or newer )
  • -L Remote site redirected to other download page to follow that use this
  • -A User agent
share|improve this answer
    
@terdon yes tested.. paste.ubuntu.com/6138009 –  Rahul Patil Sep 21 '13 at 19:25
    
My comment was left before you edited your post, the original did not work, this one does and I'd already upvoted :). –  terdon Sep 21 '13 at 19:27
    
@terdon thank you very much :) –  Rahul Patil Sep 21 '13 at 19:29
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.