I'm trying to get a site with wget. The problem is that it:

  1. Have user friendly name for the pages

  2. For some pages it uses query strings:

  3. and for some reason maybe backup some files have an extension of .1 (number from 1 - n) for example styles.css.1, javascrip.js.2

I want to do a recursive download and store it in one folder but avoid the files with queries; Home?query – in this case Home. For this I've tried --reject with a pattern but I can't make it work.

I can avoid extension .1,.2,... .n if I add a long list of numbers, but there is hopefully a better way.

This is the wget:

wget \
--page-requisites \
--no-parent \
--no-host-directories \
--no-directories \
--convert-links \
--load-cookies wget_cookies.txt --cookies=on --keep-session-cookies \
-P WikiFolders/pages/ \
-e robots=off \
-r \
--reject='1,2,3,4,5,6,7,8,9,*\\?*' \
--content-disposition \
--no-check-certificate \

If I run it this way I do get the site but is really slow for some content.

Note: to avoid files with queries I tried *\\?* but this doesn't work.

After I download I remove the content with:

find WikiFolders/pages/ -iname "*\\?*" -delete

but again this is really slow and I have to download lots of content.

I've thought on first exclude the HTML and download images, CSS and js and with other wget the HTML but since the files have no extension this doesn't work.


You can use the [] notation to specify ranges of numbers and letters. Repeat for multiple.

   |       |              |
   |       |              +---------- Reject ending with 000 to 999
   |       +------------------------- Reject ending with 00 to 99
   +--------------------------------- Reject ending with 0 to 9

This can also be used with --accept.

For the query links there is no way to filter it out – however if you specify *\?* the files will be deleted after they have been downloaded. So you would have to live with it using bandwidth and time for downloading, but you do not have to do a cleanup afterwards.

So, summa summarum, perhaps something like this:


If this does not suffice you would have to look into other tools like the one mentioned in possible duplicate link under your question.


You can avoid downloading the query links with the --reject-regex "(.*)\?(.*)" option, as explained at Using wget but ignore url parameters - Stack Overflow

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.