0

I have been trying to download specific pages in website. The site uses common URL to go to next pages like below.

https://example.com/pages/?p=1

https://example.com/pages/?p=2

https://example.com/pages/?p=3 upto 450.

I just want to download those pages and not the hyperlinks that are linked with in the pages (mean not the child pages, just parent file e.g. ?p1 and ?p2 etc).

I have tried using below command, but it is not working.

 wget --load-cookies=cookies.txt https://example.com/pages/\?p\=\{1..450\}

Is that mean {..} will not work in wget? If not, are there any option in wget that I can use to achive my goal?

5
  • If you don't want to retrieve documents recursively, then why do you use -r (and --recursive)? You also double up -np with --no-parent (these are the same option). You additionally have no space between the last option and the URLs, and the = is missing after p in the URL.
    – Kusalananda
    Apr 16, 2018 at 10:54
  • Oops. Mistakenly I have shared another try that I have used before. Updating...
    – smc
    Apr 16, 2018 at 11:02
  • @Kusalananda thanks for notifying the error. I have updated my script. Thanks in advance.
    – smc
    Apr 16, 2018 at 11:04
  • You can't escape the { or }. The shell would not recognize it as a brace expansion if you do. Just use "https://example.com/pages/?p="{1..450} (provided that wget understands how to get multiple URLs).
    – Kusalananda
    Apr 16, 2018 at 11:13
  • @Kusalananda Tried that too. Didn't work. However, I highly appreciate your help. Finally I found that solution, while loop did that work for me. Thanks.
    – smc
    Apr 16, 2018 at 11:59

3 Answers 3

1

Using a shell that understand arithmetic ranges in brace expansions (e.g. bash and ksh93 and zsh):

wget --load-cookies=cookies.txt "https://example.com/pages/?p="{1..450}

This would be expanded (before wget is called) to

wget --load-cookies=cookies.txt "https://example.com/pages/?p="1 "https://example.com/pages/?p="2 "https://example.com/pages/?p="3 ... "https://example.com/pages/?p="450

With curl:

curl --cookie-jar cookies.txt "https://example.com/pages/?p="{1..450}

Saving the output into individual files with curl:

curl --cookie-jar cookies.txt -Ooutfile{1..450}.html "https://example.com/pages/?p="{1..450}
2
  • Working great. Previously I tried quoting entire URL e.g. "example.com/pages/?p={1..450}", after seeing your answer I got to know that I have to put {..} after quoting the URL. Thanks a lot.
    – smc
    Apr 16, 2018 at 13:36
  • @smc Yes, because you need the shell to react to the brace expansion, which it wouldn't if you quoted it.
    – Kusalananda
    Apr 16, 2018 at 13:37
0

while loop does the work for me.

#!/bin/sh
a=1
while [ $a -lt 450 ]
do
wget --load-cookies=cookies.txt https://example.com/pages/\?p\=$a
a=`expr $a + 1`
done
0

Have you tried a recursive download of 1 depth from " https://example.com/pages "?

The option "-l 1" should set the recursive depth to 1.

source: https://www.gnu.org/software/wget/manual/wget.html#Recursive-Download

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.