1

There is a web server with many dxf type files which can only be downloaded one at a time. I thought it should be a good idea to use wget to download all of those files at the same time. Unfortunately, I cannot get wget to do it successfully.

My code is:

wget -r -A.dxf www.site.com/example/download

This code gives the following error: 

HTTP request sent, awaiting response... 403 Forbidden
Remote file does not exist -- broken link!!!

In case I specify the URL with exact file name

wget -O www.site.com/example/download/filename.dxf

the file downloads successfully.

How should the code look like for successfull recursive dxf file type download?

Improve this question

2 Answers 2

Reset to default
1

The 403 forbidden likely happens because the web-server forbids directory listings of the directory (thus it will not generate a file list for you). If you know the list of files you want to download, you can create a simple text file, then tell wget to download the list.

Example:

$ cat list.txt 
file1.dxf
file2.dxf
fileA.dxf

$ wget --base http://www.site.com/example/download --input-file list.txt

Will download the URLs:

Improve this answer
1
  • Ok, this helps if a site contains only a download folder. What if download folder has subfolders which names one cannot identify in advance? Does the 403 error mean that wget will not reach those subfolders unless I specify them in URL?
    – John
    Dec 30, 2015 at 4:04
1

*Given that you know the filenames, they are in sequence and the webserver lets you download in a single session *, you have a chance to get this working with shell expansion.

Suppose in your example that the filenames are file1.dxf, file2.dxf ... file5.dxf.

you can download the files as follows: 

wget -r -np -nd http://www.site.com/example/download/file{1..5}.dxf

Files downloaded will be:

  • file1.dxf
  • file2.dxf
  • file3.dxf
  • file4.dxf
  • file5.dxf

Similarly, if your files are filea.dxf, fileb.dxf .... filef.dxf

wget -r -np -nd http://www.site.com/example/download/file{a..f}.dxf

Files downloaded will be:

  • filea.dxf
  • fileb.dxf
  • filec.dxf
  • filed.dxf
  • filee.dxf
  • filef.dxf

-r: recursive, -np: no-parent, -nd: no-directories, which means that the files are downloaded but the directory hierarchy isn't.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .