2

I am trying to crawl a website for fetching pages using an API. When I issue the command

cd desktop/mysite ; curl -O https://api.mysite.com/info?page=2

I get the data on the desktop. Then I wanted to loop for pages say 2 to 100. I am unable to get the json files on to my desktop. Can someone correct my code?

for ((i=1;i<=100;i++)); do 
    cd desktop/mysite
    curl -O https://api.mysite.com/info?page=i
done
5
  • 2
    Use page=$i instead of page=i.
    – jw013
    Commented Nov 6, 2014 at 20:08
  • 2
    According to the curl manual, you can do this using without a for loop, just curl: cd desktop/mysite && curl --remote-name-all "https://api.mysite.com/info?page=[1-100]"
    – jw013
    Commented Nov 6, 2014 at 20:49
  • I need to curl with an Oauth authentication in that case how do i specify with a token curl -O "api.mysite.com/info?page=[1=100]&accesstoken=xyz" because it is not taking the token Commented Nov 11, 2014 at 20:45
  • 1
    We can't help you because we have no idea what site you are using or how it works.
    – jw013
    Commented Nov 11, 2014 at 20:47
  • I am not familiar with how that site works. See what you can find out by yourself and then post a separate question if you still need help.
    – jw013
    Commented Nov 11, 2014 at 21:03

2 Answers 2

6

You can also let curl iterate over a numeric range like this:

curl -O 'https://api.mysite.com/info?page=[1-100]'
2

You need to use $i instead of i. The former gets replaced with the value of i, while the latter is mere text.

cd desktop/mysite # If this is invariant then it does not need to be in the loop
for ((i=1;i<=100;i++)); do
  curl -O "https://api.mysite.com/info?page=$i"
done
1
  • Nice invariant comment... Commented Nov 6, 2014 at 22:14

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .