I'm trying to get a site with
wget. The problem is that it:
Have user friendly name for the pages
http://domain/wiki/Section/Home, http://domain/wiki/Section/Not+Home http://domain/wiki/Section/Other+page
For some pages it uses query strings:
and for some reason maybe backup some files have an extension of
.1(number from 1 - n) for example
I want to do a recursive download and store it in one folder but avoid the files with queries;
Home?query – in this case
Home. For this I've tried
--reject with a pattern but I can't make it work.
I can avoid extension
.1,.2,... .n if I add a long list of numbers, but there is hopefully a better way.
This is the
wget \ --page-requisites \ --no-parent \ --no-host-directories \ --no-directories \ --convert-links \ --load-cookies wget_cookies.txt --cookies=on --keep-session-cookies \ -P WikiFolders/pages/ \ -e robots=off \ -r \ --reject='1,2,3,4,5,6,7,8,9,*\\?*' \ --content-disposition \ --no-check-certificate \ http://domain/wiki/Section/
If I run it this way I do get the site but is really slow for some content.
Note: to avoid files with queries I tried
*\\?* but this doesn't work.
After I download I remove the content with:
find WikiFolders/pages/ -iname "*\\?*" -delete
but again this is really slow and I have to download lots of content.
I've thought on first exclude the
HTML and download images, CSS and js and with other wget the HTML but since the files have no extension this doesn't work.