I have tried the following:

 wget -nd -r -l1 -A gz http://www.example.com/products

This only saved a .txt file where it says:

User-agent: *
Disallow: /

On the other hand

 wget -r http://www.example.com/products/*.gz

simply does not work.

What are some other commands I should try?

  • 1
    The only acceptable solution is one that the website owner would agree to, which is why you should contact them if your request is legitimate. – Julie Pelletier Jan 2 '17 at 15:50
  • @JuliePelletier using wget to download one .gz file is fine so I assumed batch downloading should work too – dwuuuu Jan 2 '17 at 15:51
  • 1
    The server is probably refusing to give you a directory listing. – Julie Pelletier Jan 2 '17 at 15:54
  • 2
    In addition to what @JuliePelletier said, HTTP doesn't even really have the concept of a directory listing, not like FTP (or Gopher or other older protocols) does. The fact that something/foo might return an HTML page containing a list or files in the "foo" directory is only a convention. – Celada Jan 2 '17 at 15:58

Your first command does not look bad, but like others already wanted to say: the website owner placed a robots.txt which wants any search engine – or similar web spider programs, which includes wget – to stay off her site (be it to prevent unnecessary traffic or because of any other reason). wget does respect this by default, see the wget manual (e.g. info wget) section 9.1 (in Chapter 9: Appendices) for finding the tunable robot related features for your .wgetrc (also settable via -e … command line option).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.