I am using the following wget command and it downloads the required files I need except for one thing...

wget -U "Mozilla/5.0" --wait=3 --load-cookies cookies.txt --timestamping --recursive --level=2 --convert-links --no-parent --page-requisites --adjust-extension --max-redirect=0 --exclude-directories=blog --reject "*per_page=18.html" --reject "*per_page=36.html" (url here)

I want to download files like these:


But I don't want to download files like these:


I cannot seem to figure out how to reject downloading the html pages containing the extra stuff at the end.

The main problem is that wget gets stuck retrying and times out on the second types of links because the don't go anywhere - and then wget client gets banned.

Any suggestions?

2 Answers 2


Try using the --reject-regex switch of wget. You could probably do something like:

wget --recursive --no-parent --reject-regex '[^?]' url
  • I tried adding --reject-regex '[^?]' and even --reject-regex "per_page" and I can't get the switch to work at all - says something wrong with wget usage. Jun 20, 2016 at 3:23
  • Shouldn't the expression be just \?, so as to say "reject any URL containing a question mark"? The backslash is needed because ? is a regular expression meta character. Jun 20, 2016 at 5:54
  • Both do work for me whereby '\?' needs to be quoted depending on the configuration of your shell (dash needs it). What is the output of wget? Is it complaining about the regular expression or something else? Btw do you want to strip off the part after the question mark or want to omit these links entirely?
    – gabriel
    Jun 20, 2016 at 7:38
  • I'm using wget 1.13.4 on Ubuntu 12.04. command works without the --reject-regex line - but with any --reject-regex line I get the following: Usage: wget [OPTION]... [URL]... Try `wget --help' for more options. I can't seem to get the --reject option to work at all. Not sure what I'm doing wrong - I'm looking at some regex tutorials and I'm stuck. Jun 20, 2016 at 13:06
  • I think I just answered my own question - it seems --reject-regex is not in wget 1.13.4 D'Oh! Jun 20, 2016 at 13:19

What I would do, pragmatic approach ahead:

wget ....
rename 's/\.html\?.*/.html/' *.html*

This the Perl's rename command

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .