28

I have a config file which consists of list of URIs I want to download. For example,

  http://xyz.abc.com/Dir1/Dir3/sds.exe
  http://xyz.abc.com/Dir2/Dir4/jhjs.exe
  http://xyz.abc.com/Dir1/itr.exe

I want to read the config file and and copy each URL but at the same time create the same directory structure as on the host. For example, for the first line in the config file, I want to create the directory structure Dir1/Dir3 on my local machine (if it doesn't exist) and then copy sds.exe to .../Dir1/Dir3/

I found that I can download all the URLs in a file using 'wget -i' but how can I create the corresponding directory structure with that

Improve this question
33

From man wget:

-x, --force-directories:

[...]

create a hierarchy of directories, even if one would not have been created otherwise. E.g. wget -x http://fly.srk.fer.hr/robots.txt will save the downloaded file to fly.srk.fer.hr/robots.txt.

Improve this answer
5
  • Thanks for the man recommendation. I should get used to helping myself. – NGambit Apr 12 '13 at 21:16
  • Chris: But seems like if I have fly.srk.fer.hr/dir1/robots.txt then wget -x creates fly.srk.fer.hr directory and puts file named dir1/robots.txt in it. What I need it that it create even the subdirectories. i.e. save robots.txt to fly.srk.fer.hr/dir1/ – NGambit Apr 18 '13 at 14:20
  • @NGambit It's impossible to create a single file named dir1/robots.txt on Unix. – Chris Down Apr 18 '13 at 14:32
  • You are right. The config file I was using had '\' instead of '/' (since it was parsed from another windows based file) in directory paths which was messing things up. Just replaced the '\' with '/' and "wget -x -i filename" works like a charm – NGambit Apr 18 '13 at 23:22
  • 3
    A side note: sometimes you want to skip the first N folders in the URL, then you add: --cut-dirs=N – tokland Mar 3 '16 at 10:23
25

To get the structure you are asking for, I would suggest using -nH as well as -x.

This will remove the hostname as well as create the expected directory structure.

e.g.

wget -x -nH http://xyz.abc.com/Dir1/Dir3/sds.exe

- 'Dir1/Dir3/sds.exe' saved [1234]

From the man page:

-nH
--no-host-directories
   Disable generation of host-prefixed directories.  By default, invoking Wget with -r http://fly.srk.fer.hr/ will create a structure of directories beginning with fly.srk.fer.hr/.  This option disables such behavior.

-x
--force-directories
   ...create a hierarchy of directories, even if one would not have been created otherwise...
Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.