0

I have been trying to download specific pages in website. The site uses common URL to go to next pages like below.

https://example.com/pages/?p=1

https://example.com/pages/?p=2

https://example.com/pages/?p=3 upto 450.

I just want to download those pages and not the hyperlinks that are linked with in the pages (mean not the child pages, just parent file e.g. ?p1 and ?p2 etc).

I have tried using below command, but it is not working. 

 wget --load-cookies=cookies.txt https://example.com/pages/\?p\=\{1..450\}

Is that mean {..} will not work in wget? If not, are there any option in wget that I can use to achive my goal?

Improve this question
5
  • If you don't want to retrieve documents recursively, then why do you use -r (and --recursive)? You also double up -np with --no-parent (these are the same option). You additionally have no space between the last option and the URLs, and the = is missing after p in the URL. – Kusalananda♦ Apr 16 '18 at 10:54
  • Oops. Mistakenly I have shared another try that I have used before. Updating... – smc Apr 16 '18 at 11:02
  • @Kusalananda thanks for notifying the error. I have updated my script. Thanks in advance. – smc Apr 16 '18 at 11:04
  • You can't escape the { or }. The shell would not recognize it as a brace expansion if you do. Just use "https://example.com/pages/?p="{1..450} (provided that wget understands how to get multiple URLs). – Kusalananda♦ Apr 16 '18 at 11:13
  • @Kusalananda Tried that too. Didn't work. However, I highly appreciate your help. Finally I found that solution, while loop did that work for me. Thanks. – smc Apr 16 '18 at 11:59
1

Using a shell that understand arithmetic ranges in brace expansions (e.g. bash and ksh93 and zsh):

wget --load-cookies=cookies.txt "https://example.com/pages/?p="{1..450}

This would be expanded (before wget is called) to

wget --load-cookies=cookies.txt "https://example.com/pages/?p="1 "https://example.com/pages/?p="2 "https://example.com/pages/?p="3 ... "https://example.com/pages/?p="450

With curl:

curl --cookie-jar cookies.txt "https://example.com/pages/?p="{1..450}

Saving the output into individual files with curl:

curl --cookie-jar cookies.txt -Ooutfile{1..450}.html "https://example.com/pages/?p="{1..450}
Improve this answer
2
  • Working great. Previously I tried quoting entire URL e.g. "example.com/pages/?p={1..450}", after seeing your answer I got to know that I have to put {..} after quoting the URL. Thanks a lot. – smc Apr 16 '18 at 13:36
  • @smc Yes, because you need the shell to react to the brace expansion, which it wouldn't if you quoted it. – Kusalananda♦ Apr 16 '18 at 13:37
0

while loop does the work for me. 

#!/bin/sh
a=1
while [ $a -lt 450 ]
do
wget --load-cookies=cookies.txt https://example.com/pages/\?p\=$a
a=`expr $a + 1`
done
Improve this answer
0

Have you tried a recursive download of 1 depth from " https://example.com/pages "?

The option "-l 1" should set the recursive depth to 1.

source: https://www.gnu.org/software/wget/manual/wget.html#Recursive-Download

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.