wget has such option as -np which disables getting files from any parent directory. I need something similar but a bit more flexible. Consider:


I would like to get everything but not "higher" (in the tree hierarchy) than bar2 (!). So bar2 should also be fetched but not bar1.

Is there a way to make wget more selective?

Background: I'm trying to mirror a website, with a similar logical structure -- starting point, then up, then down. If there is another tool than wget, better suited for such layout, please let me know as well.


Or instead of specifying possible depth up, maybe something like "no parents, unless they match this or that URL".

Update 2

There is some structure on the server, right? You can visualize it as a tree. So normally with "--no-parent" you start from some point A and go only down.

My wish, is ability to go up -- expressed by saying, it is allowed to go up X nodes, or (which is 100% equivalent) that it is allowed to go up to B node (where the distance B-A=X).

In all cases, the rules for going down stays as were defined by users (for examples -- go down only by Y levels).

How to store it? Actually it is not the question really -- wget by default recreates the server structure, there is nothing here to be afraid, or there is no need for fixing anything. So, in 2 words -- as usual.

Update 3

Directory structure below -- let's assume that in each directory there is only one file, in R -- R.html and so on. This is simplified of course because you can have more than one page.

       / \
      B   G
     / \
    C   F
   / \
  A   D

A (A.html) is my starting point, X = 2 (so B is the most top level node I would like to fetch). In this particular example this means fetching all pages except R.html and G.html. A.html is called "starting point" because I have to start from it, not from B.

Update 4

Naming is used from Update 3.

wget OPTIONS www.foo.com/B/C/A/A.html

The question is what are the options to get all pages from directory B and below (knowing that you have to start from A.html).

  • You want bar2 fetched but not bar1? Where is bar2 going to reside? What if two or more dirs that you don't want have identically-named subdirs, should their contents be merged? It is almost certainly easier to just get the whole damn site and then prune/move things around as you desire. Commented Dec 15, 2011 at 15:21
  • @Kilian Foth, What do you mean by "get whole damn site"? Fetching it? In general it is overkill, it could mean fetching TBs when MBs are needed. For the rest, see update2. Commented Dec 15, 2011 at 15:36
  • Not sure what you mean. The only interprestation I can come up with is, you want the bar2 directory and all its contents. If that is not it, please clarify. Commented Dec 15, 2011 at 18:11
  • @Faheem Mitha, "its content" = "entire subtree". Yes, this is only interpretation I believe, and that is exactly what I mean. Commented Dec 15, 2011 at 20:58

4 Answers 4


I haven't tried it, but using -I and -X could give you what you want. My first tries would be along the line of

wget -m -I bar1/bar2 -X "*" http://www.foo.com/bar1/bar2/bar3/index.html

Explanation of options:

       Turn on options suitable for mirroring.  This option turns on recursion and time-stamping, sets
       infinite recursion depth and keeps FTP directory listings.  It is currently equivalent to -r -N -l
       inf --no-remove-listing.
-I: list
       Specify a comma-separated list of directories you wish to follow when downloading.  Elements of
       list may contain wildcards.
-X: list
       Specify a comma-separated list of directories you wish to exclude from download.  Elements of list
       may contain wildcards.

You need to add a final / to the URL, else you'll not get what you want.

If you wanted to get all content at www.myhostname.com/somedirectory then the syntax should read like:

wget -r -nH http://www.myhostname.com/somedirectory/

Try it without the end / and see what happens. Then try it with the /.

  • 1
    It will still ascend into higher directories if pages linked therein refer to such
    – EkriirkE
    Commented Oct 12, 2015 at 2:03
  • Thanks a lot for the hint with trailing slash! It helped me to solve the issue with irrelevant files fetched by wget from neighbouring directories (siblings).
    – AntonK
    Commented Jan 28, 2018 at 19:29

I think the right answer here is the --no-parent option:

       Do not ever ascend to the parent directory when retrieving recursively.
       This is a useful option, since it guarantees that only the files below
       a certain hierarchy will be downloaded.
  • This flag is the exact opposite of what OP wants
    – EkriirkE
    Commented Oct 12, 2015 at 2:02

Maybe I'm missing something, but if that is what you want then

wget -c -np -r www.foo.com/bar1/bar2

works for me (using your example). Of course, with those options you'll get all the directory structure above that too, from www.foo.com on down. If you just want bar2 at top level, then do

wget -c -np -r -nH --cut-dirs=1 www.foo.com/bar1/bar2

-nH gets rid of the www.foo.com, and --cut-dirs=1 gets rid of bar1, so you'll get bar2 and its subdirectories downloaded to the current directory. For further information, see man wget, which is quite readable and has examples.

  • You omitted starting point, you have to follow the links. You assumed that starting point is at the same time top-level (this is trivial case of np), but I am looking for general solution, when top-level is above starting point. Commented Dec 16, 2011 at 7:50
  • @macias: Sorry, I'm not following you. Can you illustrate with an example? Commented Dec 16, 2011 at 8:05
  • I just added an ASCII "screenshot". I hope this will help. In this example A is the starting point. Commented Dec 16, 2011 at 8:19
  • @macias: So you don't want to specify the path to B (as per your example), but rather A? If so, why? Is this because you want to automate some script or for some other reason? I'm also not sure what you mean by X=2. Does that mean level 2? If you are trying to fetch directories further down in the tree, I'm not sure how you distinguish B from G. Commented Dec 16, 2011 at 8:26
  • A is the starting point, because it is starting point -- look, I am on client side, not a server. IOW -- I do NOT own the server, and I didn't make this structure. I have to deal with what I see. X is symbol from Update 2, the "depth" how many levels you can go up. You distinguish B from G, because B is B, and G is not B, and you see B, because it is part of URL for A. I rephrased the question in Update 4. Commented Dec 16, 2011 at 8:49

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .