1

I am using the following wget command and it downloads the required files I need except for one thing...

wget -U "Mozilla/5.0" --wait=3 --load-cookies cookies.txt --timestamping --recursive --level=2 --convert-links --no-parent --page-requisites --adjust-extension --max-redirect=0 --exclude-directories=blog --reject "*per_page=18.html" --reject "*per_page=36.html" (url here)

I want to download files like these:

a1546997.html

But I don't want to download files like these:

a1546997.html?pwd=&per_page=36.html

I cannot seem to figure out how to reject downloading the html pages containing the extra stuff at the end.

The main problem is that wget gets stuck retrying and times out on the second types of links because the don't go anywhere - and then wget client gets banned.

Any suggestions?

0

Try using the --reject-regex switch of wget. You could probably do something like:

wget --recursive --no-parent --reject-regex '[^?]' url
  • I tried adding --reject-regex '[^?]' and even --reject-regex "per_page" and I can't get the switch to work at all - says something wrong with wget usage. – speld_rwong Jun 20 '16 at 3:23
  • Shouldn't the expression be just \?, so as to say "reject any URL containing a question mark"? The backslash is needed because ? is a regular expression meta character. – Ralph Rönnquist Jun 20 '16 at 5:54
  • Both do work for me whereby '\?' needs to be quoted depending on the configuration of your shell (dash needs it). What is the output of wget? Is it complaining about the regular expression or something else? Btw do you want to strip off the part after the question mark or want to omit these links entirely? – gabriel Jun 20 '16 at 7:38
  • I'm using wget 1.13.4 on Ubuntu 12.04. command works without the --reject-regex line - but with any --reject-regex line I get the following: Usage: wget [OPTION]... [URL]... Try `wget --help' for more options. I can't seem to get the --reject option to work at all. Not sure what I'm doing wrong - I'm looking at some regex tutorials and I'm stuck. – speld_rwong Jun 20 '16 at 13:06
  • I think I just answered my own question - it seems --reject-regex is not in wget 1.13.4 D'Oh! – speld_rwong Jun 20 '16 at 13:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.