Consider a website www.music.com with the following directory structure:

     /Franz Liszt
        Love Dream.html



I only want to get the index.html files from the nested directory in music.com/piano/covers where the name of the subdirecory starts with "Fr". In example above, I only want to download 2 files :

  • www.music.com/piano/covers/Franz Listz/ index.html
  • www.music.com/piano/covers/Frodo/ index.html

Using wget, I thought of using the following:

$ wget 
   --header="Accept: text/html"  

I executed the same on my own website, but I only get one incorrect file:


Why did I use the options above?

  1. Using --recursive or -r is not the problem as the error still persists. Furthermore, the option --page-requisites is much better alternative as I am not demanding every piece of information from the server.
  2. --domains : Ensures that nothing outside of the specified URL is downloaded. This should be the case as I don't need any resource outside the piano/covers folder
  3. --header: I want to override the Accept : * / * to prevent my request from demanding everything
  4. --html-extensions: Only download html files

It seems like the --accept-regex part isn't even considered for some reason. But it is recommended over -A option as the files I need are spanned across different directories. Any ideas how I can get the 2 files as specified by the --accept-regex argument?


Accessing the URL used in the example above gives a 404 error. So I will give context to my website which I am actually trying to perform this operation

From www.ajayhalthor.com, with the directory structure:



From this structure, I want to retrieve the files in www.ajayhalthor.com/piano that start with "Sk". I expect to retrieve the following files:

  • www.ajayhalthor.com/piano/skillet-hero
  • www.ajayhalthor.com/piano/skillet-the-last-night
  • www.ajayhalthor.com/piano/skillet-comatose
  • www.ajayhalthor.com/piano/skillet-falling-inside-the-black

Running the following command:

$ wget 
   --header="Accept: text/html"  

I get the following output:

Resolving www.ajayhalthor.com (www.ajayhalthor.com)...
Connecting to www.ajayhalthor.com (www.ajayhalthor.com)||:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘www.ajayhalthor.com/index.html’

www.ajayhalthor.com/index.html              [ <=>                                                                          ]  24.28K  --.-KB/s    in 0.1s    

Last-modified header missing -- time-stamps turned off.
2017-01-19 01:56:11 (245 KB/s) - ‘www.ajayhalthor.com/index.html’ saved [24862]

FINISHED --2017-01-19 01:56:11--
Total wall clock time: 1.4s
Downloaded: 1 files, 24K in 0.1s (245 KB/s)
Converting links in www.ajayhalthor.com/index.html... 11-1
Converted links in 1 files in 0.003 seconds.

Only 1 file www.ajayhalthor.com/index.html is downloaded.Am I using --accept-regex correctly?

  • Have you tried putting the regex in "s to prevent shell expansion? Also, try it without the .eading /: --accept-regex="piano/covers/Fr.*/index.html" – Gypsy Spellweaver Jan 18 '17 at 19:31
  • I added the quotations and eliminated the preceding slash, but the only thing that happens is the www.music.com/index.html file downloads (same output as in question) – Ajay H Jan 18 '17 at 19:38
  • Trying to access the files directly, in browser or wget returns 404 error, for all the listed index.html in your Q. Are the files there? And accessible? – Gypsy Spellweaver Jan 18 '17 at 20:00
  • I didn't disclose the true documents I am trying to access. Like I said, I tried the equivalent for my website. Should I modify my question for my exact website? – Ajay H Jan 18 '17 at 20:10
  • Either the exact website, or some sanitized copy you set up for testing. Can't see whats not working when it's not there to work with. – Gypsy Spellweaver Jan 18 '17 at 20:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.