4

I'm trying to download some directories from an Apache server, but I need to ignore some directories that have huge files I don't care about

The dir structure in the server is somewhat like this (simplified):

somedir/
├── atxt.txt
├── big_file.pdf
├── image.jpg
└── tmp
    └── tempfile.txt

So, I want to get all the .txt and .jpg files, but I DON'T want the .pdf files nor anything that is in a tmp directory.

I've tried using --exclude-directories together with --accept and then with --reject, but in both attempts it keeps downloading the tmp dir and its contents.

These are the commands I've tried:

# with --reject
wget -nH --cut-dirs=2 -r --reject=pdf --exclude-directories=tmp \
         --no-parent  http://<host>/pub/somedir/

# with --accept
wget -nH --cut-dirs=2 -r --accept=txt,jpg --exclude-directories=tmp \
         --no-parent  http://<host>/pub/somedir/

Is there a way to do this?

How exactly is --exclude-directories supposed to work?

3 Answers 3

4

Rather than try and do this using wget I'd suggest using a more appropriate tool for downloading complex "sets" of files or filters.

You can use httrack to download either entire directories of files (essentially mirror everything from a site) or you can specify to httrack a filter along with specific file extensions, such as download only .pdf files.

You can read more about httrack's filter capability which is what you'd need to use if you were interested in only downloading files that were named in a specific way.

Here are some examples of the wildcard capability:

  • *[file] or *[name] - any filename or name, e.g. not /,? and ; characters
  • *[path] - any path (and filename), e.g. not ? and ; characters
  • *[a,z,e,r,t,y] - any letters among a,z,e,r,t,y
  • *[a-z] - any letters
  • *[0-9,a,z,e,r,t,y] - any characters among 0..9 and a,z,e,r,t,y

Example

$ httrack http://url.com/files/ -* +1_[a-z].doc -O /dir/to/output

The switches are as follows:

  • -* - remove everything from list of things to download
  • +1_[a-z].doc - download files named 1_a.doc, 1_b.doc, etc.
  • -O /dir/to/output - write results here
6
  • httrack is definitely a better way.
    – Kiwy
    Feb 7, 2014 at 15:28
  • httrack -W always recommended.
    – user55518
    Feb 7, 2014 at 15:32
  • 1
    Thanks for your answer, I'm going to give httrack a try. I really wish to know how the --exclude-directories option works, though. It's ignored even without combining --accept or --reject.
    – elias
    Feb 7, 2014 at 16:18
  • 1
    @elias - in the man page it says it takes wildcards, so perhaps you need to define the "directories" using something like */tmp/*.
    – slm
    Feb 7, 2014 at 17:15
  • 1
    httrack does not support custom headers (needed for authentication). Wget does. Oct 22, 2014 at 9:43
0

I had a similar case where I did the equivalent of what would be for you --exclude-directories '/tmp' and it worked for me.

See also https://serverfault.com/questions/156045/how-to-download-with-wget-without-following-links-with-parameters/156057#156057 for more powerful options on recent wget (regular expressions).

0

Many including me had problems figuring out how to use -X, and found the answer hard to remember (between years of span between use).

This is a hard to remember trick write it down :)

$ wget -X fo*o ... # does not work for directories
$ wget -X */fo*o,*/*/fo*o,*/*/*/fo*o ... # works for directories
$ wget -X \*/fo\*o/,\*/\*/fo\*o/,\*/\*/\*/fo\*o/ ...

ANSWER:

site hack wget-?/src/utils.c so to use basename instead of current path, also on command line use filename pattern (just 'fo*o', nothing else). and note param FNM_PATHNAME set to 0 (otherwise it demands '/' to finish tail of match). the following patch is against wget-1.12 there my be a simpler way by definining ?FNM_FLAGS but this works "fine".

--- utils.c.old 2016-09-13 07:49:11.000000000 -0400
+++ utils.c 2016-09-13 09:32:58.000000000 -0400
@@ -907,6 +907,9 @@
   return *d1 == '\0' && (*d2 == '\0' || *d2 == '/');
 }

+/* for basename */
+#include <libgen.h>
+
 /* Iterate through DIRLIST (which must be NULL-terminated), and return the
    first element that matches DIR, through wildcards or front comparison (as
    appropriate).  */
@@ -921,18 +924,24 @@
     {
       /* Remove leading '/' */
       char *p = *x + (**x == '/');
+      /* SITE HACK - only if patterned ignore leading dirs cmp as file */
+      char sh_str[1024*16], *pp;
+      strcpy(sh_str,basename(dir));
+      pp=sh_str;
+#if 0
+      printf("? %s == %s ?\n",p,pp); 
+#endif
       if (has_wildcards_p (p))
         {
-          if (matcher (p, dir, FNM_PATHNAME) == 0)
+          if (matcher (p, pp, 0) == 0)
             break;
         }
       else
         {
-          if (subdir_p (p, dir))
+          if (subdir_p (p, pp))
             break;
         }
     }
-
   return *x ? true : false;
 }

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .