40

I have a config file which consists of list of URIs I want to download. For example,

  http://xyz.abc.com/Dir1/Dir3/sds.exe
  http://xyz.abc.com/Dir2/Dir4/jhjs.exe
  http://xyz.abc.com/Dir1/itr.exe

I want to read the config file and and copy each URL but at the same time create the same directory structure as on the host. For example, for the first line in the config file, I want to create the directory structure Dir1/Dir3 on my local machine (if it doesn't exist) and then copy sds.exe to .../Dir1/Dir3/

I found that I can download all the URLs in a file using 'wget -i' but how can I create the corresponding directory structure with that

2 Answers 2

47

From man wget:

-x, --force-directories:

[...]

create a hierarchy of directories, even if one would not have been created otherwise. E.g. wget -x http://fly.srk.fer.hr/robots.txt will save the downloaded file to fly.srk.fer.hr/robots.txt.

6
  • Thanks for the man recommendation. I should get used to helping myself.
    – NGambit
    Apr 12, 2013 at 21:16
  • Chris: But seems like if I have fly.srk.fer.hr/dir1/robots.txt then wget -x creates fly.srk.fer.hr directory and puts file named dir1/robots.txt in it. What I need it that it create even the subdirectories. i.e. save robots.txt to fly.srk.fer.hr/dir1/
    – NGambit
    Apr 18, 2013 at 14:20
  • @NGambit It's impossible to create a single file named dir1/robots.txt on Unix.
    – Chris Down
    Apr 18, 2013 at 14:32
  • 7
    A side note: sometimes you want to skip the first N folders in the URL, then you add: --cut-dirs=N
    – tokland
    Mar 3, 2016 at 10:23
  • 1
    Also, for ignoring the hostname directory add -nH besides --cut-dirs=N flag. Nov 3, 2021 at 12:37
38

To get the structure you are asking for, I would suggest using -nH as well as -x.

This will remove the hostname as well as create the expected directory structure.

e.g.

wget -x -nH http://xyz.abc.com/Dir1/Dir3/sds.exe

- 'Dir1/Dir3/sds.exe' saved [1234]

From the man page:

-nH
--no-host-directories
   Disable generation of host-prefixed directories.  By default, invoking Wget with -r http://fly.srk.fer.hr/ will create a structure of directories beginning with fly.srk.fer.hr/.  This option disables such behavior.

-x
--force-directories
   ...create a hierarchy of directories, even if one would not have been created otherwise...
1
  • +1 Thanks for mentioning -nH flag. I already knew how to use --cut-dirs=N to ignore first N directories but the hostname directory was being created. Combining the 2 helped in my case. Nov 3, 2021 at 12:36

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .