Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'm trying to download some directories from an Apache server, but I need to ignore some directories that have huge files I don't care about

The dir structure in the server is somewhat like this (simplified):

├── atxt.txt
├── big_file.pdf
├── image.jpg
└── tmp
    └── tempfile.txt

So, I want to get all the .txt and .jpg files, but I DON'T want the .pdf files nor anything that is in a tmp directory.

I've tried using --exclude-directories together with --accept and then with --reject, but in both attempts it keeps downloading the tmp dir and its contents.

These are the commands I've tried:

# with --reject
wget -nH --cut-dirs=2 -r --reject=pdf --exclude-directories=tmp \
         --no-parent  http://<host>/pub/somedir/

# with --accept
wget -nH --cut-dirs=2 -r --accept=txt,jpg --exclude-directories=tmp \
         --no-parent  http://<host>/pub/somedir/

Is there a way to do this?

How exactly is --exclude-directories supposed to work?

share|improve this question

Rather than try and do this using wget I'd suggest using a more appropriate tool for downloading complex "sets" of files or filters.

You can use httrack to download either entire directories of files (essentially mirror everything from a site) or you can specify to httrack a filter along with specific file extensions, such as download only .pdf files.

You can read more about httrack's filter capability which is what you'd need to use if you were interested in only downloading files that were named in a specific way.

Here are some examples of the wildcard capability:

  • *[file] or *[name] - any filename or name, e.g. not /,? and ; characters
  • *[path] - any path (and filename), e.g. not ? and ; characters
  • *[a,z,e,r,t,y] - any letters among a,z,e,r,t,y
  • *[a-z] - any letters
  • *[0-9,a,z,e,r,t,y] - any characters among 0..9 and a,z,e,r,t,y


$ httrack http://url.com/files/ -* +1_[a-z].doc -O /dir/to/output

The switches are as follows:

  • -* - remove everything from list of things to download
  • +1_[a-z].doc - download files named 1_a.doc, 1_b.doc, etc.
  • -O /dir/to/output - write results here
share|improve this answer
httrack is definitely a better way. – Kiwy Feb 7 '14 at 15:28
httrack -W always recommended. – user55518 Feb 7 '14 at 15:32
Thanks for your answer, I'm going to give httrack a try. I really wish to know how the --exclude-directories option works, though. It's ignored even without combining --accept or --reject. – elias Feb 7 '14 at 16:18
@elias - in the man page it says it takes wildcards, so perhaps you need to define the "directories" using something like */tmp/*. – slm Feb 7 '14 at 17:15
httrack does not support custom headers (needed for authentication). Wget does. – Stéphane Gourichon Oct 22 '14 at 9:43

I had a similar case where I did the equivalent of what would be for you --exclude-directories '/tmp' and it worked for me.

See also https://serverfault.com/questions/156045/how-to-download-with-wget-without-following-links-with-parameters/156057#156057 for more powerful options on recent wget (regular expressions).

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.