Wget command that returns [a list of] all webpages on a webserver

Question

Let's say we have a website www.example.com with the following pages:

www.example.com/page1
www.example.com/unknown
www.example.com/unknown

Is there a Wget command that would produce the following output:

page1
page2
page3

How would this command change if the website structure was:

www.example.com/xxxx/page1
www.example.com/xxxx/page2
www.example.com/xxxx/page3

Essentially I have a server ip address and would like to be able to list all the pages held on the server, to see if there are any I am interested in to download.

For instance I can do wget -r www.example.com/page1 and I want to know if there are any other directories (page2, page3, etc) that I might be interested in.

I have researched the --spider option and others, but with no joy.

Answer 1

This is not possible. There is no HTTP request method for that, a HTTP retrieve request gets always a particular URL as parameter.

Answer 2

You can tell wget to recursively download an entire website, but it does so by following the links on each page. If it doesn't know page1, page2 and page3 are there, it will never retrieve them.

Put bluntly, HTTP does not work that way -- fortunately.

Answer 3

You can't do this from the client end but you can look for a site-map, sometimes the http://www.example.com/robots.txt file might contain a list. There may be a way to ask Google for a list and there may be a last at the wayback machine.

Answer 4

Like Rens and franziskus say, there is no way to do that from page1, the only chance will depend on how the website you want to copy is set-up.

It is unlikely in the root directory, but sub-directories (providing you know they exist) may be configured in such a way that they give you a list of files (some sort of visual ftp). But you're up to exploiting what most webmasters are trying to hide away from you: the internals of their websites.

I have successfully exploited this to get to information I was confident was there, but could not find in any way through website navigation. It only works with very few websites.