0

I have been trying to download specific pages in website. The site uses common URL to go to next pages like below.

https://example.com/pages/?p=1

https://example.com/pages/?p=2

https://example.com/pages/?p=3 upto 450.

I just want to download those pages and not the hyperlinks that are linked with in the pages (mean not the child pages, just parent file e.g. ?p1 and ?p2 etc).

I have tried using below command, but it is not working.

 wget --load-cookies=cookies.txt https://example.com/pages/\?p\=\{1..450\}

Is that mean {..} will not work in wget? If not, are there any option in wget that I can use to achive my goal?

5
  • If you don't want to retrieve documents recursively, then why do you use -r (and --recursive)? You also double up -np with --no-parent (these are the same option). You additionally have no space between the last option and the URLs, and the = is missing after p in the URL.
    – Kusalananda
    Apr 16, 2018 at 10:54
  • Oops. Mistakenly I have shared another try that I have used before. Updating...
    – smc
    Apr 16, 2018 at 11:02
  • @Kusalananda thanks for notifying the error. I have updated my script. Thanks in advance.
    – smc
    Apr 16, 2018 at 11:04
  • You can't escape the { or }. The shell would not recognize it as a brace expansion if you do. Just use "https://example.com/pages/?p="{1..450} (provided that wget understands how to get multiple URLs).
    – Kusalananda
    Apr 16, 2018 at 11:13
  • @Kusalananda Tried that too. Didn't work. However, I highly appreciate your help. Finally I found that solution, while loop did that work for me. Thanks.
    – smc
    Apr 16, 2018 at 11:59

3 Answers 3

1

Using a shell that understand arithmetic ranges in brace expansions (e.g. bash and ksh93 and zsh):

wget --load-cookies=cookies.txt "https://example.com/pages/?p="{1..450}

This would be expanded (before wget is called) to

wget --load-cookies=cookies.txt "https://example.com/pages/?p="1 "https://example.com/pages/?p="2 "https://example.com/pages/?p="3 ... "https://example.com/pages/?p="450

With curl:

curl --cookie-jar cookies.txt "https://example.com/pages/?p="{1..450}

Saving the output into individual files with curl:

curl --cookie-jar cookies.txt -Ooutfile{1..450}.html "https://example.com/pages/?p="{1..450}
2
  • Working great. Previously I tried quoting entire URL e.g. "example.com/pages/?p={1..450}", after seeing your answer I got to know that I have to put {..} after quoting the URL. Thanks a lot.
    – smc
    Apr 16, 2018 at 13:36
  • @smc Yes, because you need the shell to react to the brace expansion, which it wouldn't if you quoted it.
    – Kusalananda
    Apr 16, 2018 at 13:37
0

while loop does the work for me.

#!/bin/sh
a=1
while [ $a -lt 450 ]
do
wget --load-cookies=cookies.txt https://example.com/pages/\?p\=$a
a=`expr $a + 1`
done
0

Have you tried a recursive download of 1 depth from " https://example.com/pages "?

The option "-l 1" should set the recursive depth to 1.

source: https://www.gnu.org/software/wget/manual/wget.html#Recursive-Download

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .