Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Let's say we have a website www.example.com with the following pages:


Is there a Wget command that would produce the following output:


How would this command change if the website structure was:


Essentially I have a server ip address and would like to be able to list all the pages held on the server, to see if there are any I am interested in to download.

For instance I can do wget -r www.example.com/page1 and I want to know if there are any other directories (page2, page3, etc) that I might be interested in.

I have researched the --spider option and others, but with no joy.

share|improve this question
I don't understand how you get from unknown to page2 and page3. More generally, what are you looking for? Do you want to see what wget downloaded? or what is chose not to download? – Gilles Feb 26 '11 at 19:02
I would like the wget command to return the page names that are stored in a given directory. For instance, I give the input as "example.com/" and it returns with all the pages in the root directory of the website – user5152 Feb 26 '11 at 19:45

You can't do this from the client end but you can look for a site-map, sometimes the http://www.example.com/robots.txt file might contain a list. There may be a way to ask Google for a list and there may be a last at the wayback machine.

share|improve this answer

This is not possible. There is no HTTP request method for that, a HTTP retrieve request gets always a particular URL as parameter.

share|improve this answer
up vote 3 down vote accepted

You can tell wget to recursively download an entire website, but it does so by following the links on each page. If it doesn't know page1, page2 and page3 are there, it will never retrieve them.

Put bluntly, HTTP does not work that way -- fortunately.

share|improve this answer
So when I do "wget -r www.example.com" how is the wget command finding the various pages that are part of that website? (like www.example.com/page1) – user5152 Feb 26 '11 at 21:29
it parses the contents of the html page returned by the server and looks for links <a href="(link)"> – franziskus Feb 26 '11 at 21:32

Like Rens and franziskus say, there is no way to do that from page1, the only chance will depend on how the website you want to copy is set-up.

It is unlikely in the root directory, but sub-directories (providing you know they exist) may be configured in such a way that they give you a list of files (some sort of visual ftp). But you're up to exploiting what most webmasters are trying to hide away from you: the internals of their websites.

I have successfully exploited this to get to information I was confident was there, but could not find in any way through website navigation. It only works with very few websites.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.