1

Consider a website www.music.com with the following directory structure:

/piano
   /covers
     /Chopin 
        apple.html
        bannan.js
        balloon.html
        index.html
     /Franz Liszt
        index.html
        roses.js
        Love Dream.html

     /Frodo
        index.html
        linkenso.html

/violin
   /covers
      /David
         Viva.html
      /Ross
         index.html

I only want to get the index.html files from the nested directory in music.com/piano/covers where the name of the subdirecory starts with "Fr". In example above, I only want to download 2 files :

  • www.music.com/piano/covers/Franz Listz/ index.html
  • www.music.com/piano/covers/Frodo/ index.html

Using wget, I thought of using the following:

$ wget 
   --mirror 
   --header="Accept: text/html"  
   --page-requisites 
   --html-extension 
   --convert-links 
   --restrict-file-names=windows 
   --domains=www.music.com/piano/covers 
   --accept-regex=/piano/covers/Fr.*/index.html  
        http://www.music.com

I executed the same on my own website, but I only get one incorrect file:

www.music.com/index.html

Why did I use the options above?

  1. Using --recursive or -r is not the problem as the error still persists. Furthermore, the option --page-requisites is much better alternative as I am not demanding every piece of information from the server.
  2. --domains : Ensures that nothing outside of the specified URL is downloaded. This should be the case as I don't need any resource outside the piano/covers folder
  3. --header: I want to override the Accept : * / * to prevent my request from demanding everything
  4. --html-extensions: Only download html files

It seems like the --accept-regex part isn't even considered for some reason. But it is recommended over -A option as the files I need are spanned across different directories. Any ideas how I can get the 2 files as specified by the --accept-regex argument?

EDIT 1:

Accessing the URL used in the example above gives a 404 error. So I will give context to my website which I am actually trying to perform this operation

From www.ajayhalthor.com, with the directory structure:

/piano
   /nightwish-sahara
   /nightwish-amaranth
   /skillet-hero
   /skillet-the-last-night
   /breaking-benjamin-diary-of-jane
   /skillet-comatose
   /one-republic-counting-stars
   /skillet-falling-inside-the-black
   /63/index.html
   /a/few/more/links/index.html

/about
   /other/links/index.html
/Home
   /main/links/index.html

From this structure, I want to retrieve the files in www.ajayhalthor.com/piano that start with "Sk". I expect to retrieve the following files:

  • www.ajayhalthor.com/piano/skillet-hero
  • www.ajayhalthor.com/piano/skillet-the-last-night
  • www.ajayhalthor.com/piano/skillet-comatose
  • www.ajayhalthor.com/piano/skillet-falling-inside-the-black

Running the following command:

$ wget 
   --mirror 
   --header="Accept: text/html"  
   --page-requisites 
   --html-extension 
   --convert-links 
   --restrict-file-names=windows 
   --domains=www.ajayhalthor.com/piano
   --accept-regex="piano/sk.*"
        http://www.ajayhalthor.com

I get the following output:

Resolving www.ajayhalthor.com (www.ajayhalthor.com)... 23.229.213.7
Connecting to www.ajayhalthor.com (www.ajayhalthor.com)|23.229.213.7|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘www.ajayhalthor.com/index.html’

www.ajayhalthor.com/index.html              [ <=>                                                                          ]  24.28K  --.-KB/s    in 0.1s    

Last-modified header missing -- time-stamps turned off.
2017-01-19 01:56:11 (245 KB/s) - ‘www.ajayhalthor.com/index.html’ saved [24862]

FINISHED --2017-01-19 01:56:11--
Total wall clock time: 1.4s
Downloaded: 1 files, 24K in 0.1s (245 KB/s)
Converting links in www.ajayhalthor.com/index.html... 11-1
Converted links in 1 files in 0.003 seconds.

Only 1 file www.ajayhalthor.com/index.html is downloaded.Am I using --accept-regex correctly?

  • Have you tried putting the regex in "s to prevent shell expansion? Also, try it without the .eading /: --accept-regex="piano/covers/Fr.*/index.html" – Gypsy Spellweaver Jan 18 '17 at 19:31
  • I added the quotations and eliminated the preceding slash, but the only thing that happens is the www.music.com/index.html file downloads (same output as in question) – Ajay H Jan 18 '17 at 19:38
  • Trying to access the files directly, in browser or wget returns 404 error, for all the listed index.html in your Q. Are the files there? And accessible? – Gypsy Spellweaver Jan 18 '17 at 20:00
  • I didn't disclose the true documents I am trying to access. Like I said, I tried the equivalent for my website. Should I modify my question for my exact website? – Ajay H Jan 18 '17 at 20:10
  • Either the exact website, or some sanitized copy you set up for testing. Can't see whats not working when it's not there to work with. – Gypsy Spellweaver Jan 18 '17 at 20:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.