There are a number of questions similar to this on StackExchange but none address this issue.

I want to download all the pdf files in the 2007 directory at http://www3.cs.stonybrook.edu/~algorith/video-lectures/.

So I want wget to parse the html file available at the above link and only follow links that go to pdf files in the 2007 directory.

I used the following but it didn't work:

wget -r -A pdf -I /2007 'http://www3.cs.stonybrook.edu/~algorith/video-lectures/'

Can you also explain why the above does not work?

  • 1
    You want to spider using the solution here stackoverflow.com/questions/2804467/… Then GREP the results and call WGET. – Cat Man Do Jan 17 '15 at 5:01
  • @CatManDo Thanks, I can think of a few workarounds but I also think that wget alone must be able to solve this problem. Why should my above solution not work? – Loax Jan 17 '15 at 5:09

As noted by Anthon the -I option does not work that way. But, as you have a reference point - namely ~algorith/video-lectures/ with a listing of files there are some options. One is to parse index with other tools and re-run wget. Another is to use --accept-regex: it matches for accept on the complete URL.

From man:

--accept-regex urlregex
--reject-regex urlregex
    Specify a regular expression to accept or reject the complete URL.

This should do what you want:

wget -r -nd -A pdf --accept-regex "2007/.*\.pdf" 'http://www3.cs.stonybrook.edu/~algorith/video-lectures/'

Remove -nd if you actually want the directories.

Edit (to address comment)

accept vs. accept-regex

This is somewhat cumbersome for me to explain, but I'll give it a try.

First off, if you really want to read the manual, then use info. As stated in man (this is from GNU wget) - (easy to overlook):

   This is not the complete manual for GNU Wget.  For more complete information,
   including more detailed explanations of some of the options, and a number of
   commands available for use with .wgetrc files and the -e option, see the GNU
   Info entry for wget.

In this case i.e.:

$ info wget "Following Links" "Types of Files"

or online.

Here we find, emphasize mine:

Finally, it’s worth noting that the accept/reject lists are matched twice against downloaded files: once against the URL’s filename portion, to determine if the file should be downloaded in the first place; then, after it has been accepted and successfully downloaded, the local file’s name is also checked against the accept/reject lists to see if it should be removed.

Further it continues to explain that the rationale behind this is that

  1. .htm and .html files are always downloaded regardless of accept/reject rules.
  2. They should be removed after being downloaded and scanned for links, if they did match the accept/reject lists.

Thus: HTML files are always downloaded. After it is downloaded the match is only done against the file name.

Not sure how much this helped. If you read the info page it might be more clear. It is a bit of complexity with chicken and egg things etc. in the mix here.

  • Nice. I tried doing this with -A and realised that it is only for the file portion of the link. Didn't know --accept-regex would be matched against the entire URL. Why do I still need -A pdf though? In fact without it 'index.html' is not removed after download. Would I be right to think that -A and --accept-regex are, despite the documentation, of different nature. The former determines what files will remain after download whereas the latter determines what URLs must be fetched? – Loax Jan 17 '15 at 16:37
  • @Loax: Yes, they are of different nature. Added some more information. – user367890 Jan 17 '15 at 17:59

The -I option as described in man wget:

-I list
    Specify a comma-separated list of directories you wish to follow
    when downloading.  Elements of list may contain wildcards.

You interpret this (understandably) as a filter on the URLs found on the original page, but that is wishful thinking. It is a filter on the subdirectories into which wget can descend for which and index.html can be found. As, as you already noticed yourself, there is no such index.html in http://www3.cs.stonybrook.edu/~algorith/video-lectures/2007/

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.