Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I'm not very good with Linux so I have maybe a basic question. I've search around for the "how to" and I couldn't find anything.

I'm trying to get a site with wget. The problem is that it:

  1. Have user friendly name for the pages

    http://domain/wiki/Section/Home, 
    http://domain/wiki/Section/Not+Home
    http://domain/wiki/Section/Other+page
    
  2. For some pages it uses query strings:

    http://domain/wiki/Section/Home?one=value&other=value
    
  3. and for some reason maybe backup some files have an extension of .1 (number from 1 - n) for example styles.css.1, javascrip.js.2

I want to do a recursive download and store it in one folder but avoid the files with queries; Home?query – in this case Home. For this I've tried --reject with a pattern but i cant make it work.

I can avoid extension .1,.2,... .n if I add a long list of numbers, but there is hopefully a better way.

This is the wget:

wget \
--page-requisites \
--no-parent \
--no-host-directories \
--no-directories \
--convert-links \
--load-cookies wget_cookies.txt --cookies=on --keep-session-cookies \
-P WikiFolders/pages/ \
-e robots=off \
-r \
--reject='1,2,3,4,5,6,7,8,9,*\\?*' \
--content-disposition \
--no-check-certificate \
http://domain/wiki/Section/

If i run it this way I do get the site but is really slow for some content.

Note: to avoid files with queries I tried *\\?* but this doesn't work.

After I download I remove the content with:

find WikiFolders/pages/ -iname "*\\?*" -delete

but again this is really slow and I have to download lots of content.

I've thought on first exclude the HTML and download images, css and js and with other wget the HTML but since the files have no extension this doesn't work.

I hope this is possible and someone can help me with it.

share|improve this question
    
possible duplicate: unix.stackexchange.com/questions/73367/… –  slm May 21 '13 at 21:31
add comment

1 Answer 1

up vote 1 down vote accepted

You can use the [] notation to specify ranges of numbers and letters. Repeat for multiple.

*[0-9],*[0-9][0-9],*[0-9][0-9][0-9]
|____||__________||_______________|
   |       |              |
   |       |              +---------- Reject ending with 000 to 999
   |       +------------------------- Reject ending with 00 to 99
   +--------------------------------- Reject ending with 0 to 9

This can also be used with --accept.

For the query links there is no way to filter it out – however if you specify *\?* the files will be deleted after they have been downloaded. So you would have to live with it using bandwidth and time for downloading, but you do not have to do a cleanup afterwards.

So, summa summarum, perhaps something like this:

--reject='*.[0-9],*.[0-9][0-9],*\?*'

If this does not suffice you would have to look into other tools like the one mentioned in possible duplicate link under your question.

share|improve this answer
    
It worked thanks –  cesaregb May 22 '13 at 1:26
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.