Let's say we have a website
www.example.com with the following pages:
www.example.com/page1 www.example.com/unknown www.example.com/unknown
Is there a Wget command that would produce the following output:
page1 page2 page3
How would this command change if the website structure was:
www.example.com/xxxx/page1 www.example.com/xxxx/page2 www.example.com/xxxx/page3
Essentially I have a server ip address and would like to be able to list all the pages held on the server, to see if there are any I am interested in to download.
For instance I can do
wget -r www.example.com/page1 and I want to know if there are any other directories (
page3, etc) that I might be interested in.
I have researched the
--spider option and others, but with no joy.