I'm new to using bash, and I have been trying to wget all the files from a website to the server I have been working on. However all I'm getting back is an index.html file. I let it run for 15 minutes and the index.html file was still downloading so I killed it. Might my files be download after the index.html file?

Here is the code I have been trying:

$ wget --no-parent -R index.html -A "Sample" -nd --random-wait \
   -r -p -e robots=off -U Mozilla --no-check-certificate \

I'm trying to download all the files in a subdirectory that starts with Sample. I have searched quite a bit on the internet to find a resolution, and at this point I'm stumped. I probably just haven't found the right combinations of options, but any help would be much appreciated. Here is my understanding of the code:

  • --no-parent means don't search parent directories
  • -R index.html means reject downloading the index.html file, I also tried "index.html*", but it still downloaded it anyway
  • -A "Sample" kind of acts like a Sample* would in bash
  • -nd means download the files and not any of the directories
  • --random-wait to make sure you don't get blacklisted from a site
  • -r recursively downloads
  • -p not sure really
  • -e robots=off ignores robot.txt files
  • U Mozilla makes the user look like its Mozilla I think
  • The --no-check-certificate is just necessary for the website.
  • You can start by replace: --no-check- certificate, by: --no-check-certificate. – slackmart Jul 15 '14 at 15:28
  • oops my bad. I must have added those spaces while typing it out on here. Its not like that in my code. I'll edit it momentarily – njBernstein Jul 15 '14 at 15:29
  • see this question: unix.stackexchange.com/questions/143996/… – Rabin Jul 15 '14 at 15:34

-A "Sample" kind of acts like a Sample* would in bash

Not by my reading of man wget:

  • -A acclist --accept acclist
  • -R rejlist --reject rejlist

Specify comma-separated lists of file name suffixes or patterns to accept or reject. Note that if any of the wildcard characters, *, ?, [ or ], appear in an element of acclist or rejlist, it will be treated as a pattern, rather than a suffix.

So your usage (no wildcards) is equivalent to the bash glob *.Sample.

Wget works by scanning links, which is probably why it is trying to download an index.html (you haven't said what the content of that is, if any, just that it took a long time) -- it has to have somewhere to start. To explain further: an URL is not a file path. You cannot scan a web server as if it were a directory hierarchy, saying, "give me all the files in directory foobar". Iffoobar corresponds to a real directory (it certainly doesn't have to, because it's part of an URL, not a file path), a web server may be configured to provide an autogenerated index.html listing the files, providing the illusion that you can browse the filesystem. But that's not part of the HTTP protocol, it's just a convention used by default with servers like apache. So what wget does is scan, e.g., index.html for <a href= and <img src=, etc., then it follows those links and does the same thing, recursively. That's what wget's "recursive" behaviour refers to -- it recursively scans links because (to reiterate), it does not have access to any filesystem on the server, and the server does not have to provide it with ANY information regarding such.

If you have an actual .html web page that you can load and click through to all the things you want, start with that address, and use just -r -np -k -p.


What happens when you connect to the page with your browser?

If you browse the pages without any problem, then there might be a couple of things you're missing.

The page might be checking your browser, and Mozilla is not the correct answer, pick one of the browser strings from here (the whole string, not just Mozilla) and try to see if it works.

If it doesn't then you might need cookies, get the cookies connecting to the main page with wget and store them. Then use wget with those cookies and try to download the pages.

Try --mirror option if you want to mirror it.

If nothing works, then I would study the connection and the pages. The live http headers add on for Firefox is a pretty cool tool. You can see the whole communication between your browser and the web page. Try to mimmic that behaviour with wget to obtain what you're looking for.

  • I forgot, another important thing, you might need the referer. – YoMismo Jul 15 '14 at 15:38
  • I browse the page without problem. I'll look into your suggestions and get back to you. Thanks for all of your help – njBernstein Jul 15 '14 at 15:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.