1

There is a web server with many dxf type files which can only be downloaded one at a time. I thought it should be a good idea to use wget to download all of those files at the same time. Unfortunately, I cannot get wget to do it successfully.

My code is:

wget -r -A.dxf www.site.com/example/download

This code gives the following error: 

HTTP request sent, awaiting response... 403 Forbidden
Remote file does not exist -- broken link!!!

In case I specify the URL with exact file name

wget -O www.site.com/example/download/filename.dxf

the file downloads successfully.

How should the code look like for successfull recursive dxf file type download?

2 Answers 2

Reset to default
1

The 403 forbidden likely happens because the web-server forbids directory listings of the directory (thus it will not generate a file list for you). If you know the list of files you want to download, you can create a simple text file, then tell wget to download the list.

Example:

$ cat list.txt 
file1.dxf
file2.dxf
fileA.dxf

$ wget --base http://www.site.com/example/download --input-file list.txt

Will download the URLs:

1
  • Ok, this helps if a site contains only a download folder. What if download folder has subfolders which names one cannot identify in advance? Does the 403 error mean that wget will not reach those subfolders unless I specify them in URL?
    – John
    Dec 30, 2015 at 4:04
1

*Given that you know the filenames, they are in sequence and the webserver lets you download in a single session *, you have a chance to get this working with shell expansion.

Suppose in your example that the filenames are file1.dxf, file2.dxf ... file5.dxf.

you can download the files as follows: 

wget -r -np -nd http://www.site.com/example/download/file{1..5}.dxf

Files downloaded will be:

  • file1.dxf
  • file2.dxf
  • file3.dxf
  • file4.dxf
  • file5.dxf

Similarly, if your files are filea.dxf, fileb.dxf .... filef.dxf

wget -r -np -nd http://www.site.com/example/download/file{a..f}.dxf

Files downloaded will be:

  • filea.dxf
  • fileb.dxf
  • filec.dxf
  • filed.dxf
  • filee.dxf
  • filef.dxf

-r: recursive, -np: no-parent, -nd: no-directories, which means that the files are downloaded but the directory hierarchy isn't.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .