3

I am using the following wget command and it downloads the required files I need except for one thing...

wget -U "Mozilla/5.0" --wait=3 --load-cookies cookies.txt --timestamping --recursive --level=2 --convert-links --no-parent --page-requisites --adjust-extension --max-redirect=0 --exclude-directories=blog --reject "*per_page=18.html" --reject "*per_page=36.html" (url here)

I want to download files like these:

a1546997.html

But I don't want to download files like these:

a1546997.html?pwd=&per_page=36.html

I cannot seem to figure out how to reject downloading the html pages containing the extra stuff at the end.

The main problem is that wget gets stuck retrying and times out on the second types of links because the don't go anywhere - and then wget client gets banned.

Any suggestions?

Improve this question

2 Answers 2

Reset to default
3

Try using the --reject-regex switch of wget. You could probably do something like: 

wget --recursive --no-parent --reject-regex '[^?]' url
Improve this answer
8
  • I tried adding --reject-regex '[^?]' and even --reject-regex "per_page" and I can't get the switch to work at all - says something wrong with wget usage.
    – speld_rwong
    Jun 20, 2016 at 3:23
  • Shouldn't the expression be just \?, so as to say "reject any URL containing a question mark"? The backslash is needed because ? is a regular expression meta character.
    – Ralph Rönnquist
    Jun 20, 2016 at 5:54
  • Both do work for me whereby '\?' needs to be quoted depending on the configuration of your shell (dash needs it). What is the output of wget? Is it complaining about the regular expression or something else? Btw do you want to strip off the part after the question mark or want to omit these links entirely?
    – gabriel
    Jun 20, 2016 at 7:38
  • I'm using wget 1.13.4 on Ubuntu 12.04. command works without the --reject-regex line - but with any --reject-regex line I get the following: Usage: wget [OPTION]... [URL]... Try `wget --help' for more options. I can't seem to get the --reject option to work at all. Not sure what I'm doing wrong - I'm looking at some regex tutorials and I'm stuck.
    – speld_rwong
    Jun 20, 2016 at 13:06
  • I think I just answered my own question - it seems --reject-regex is not in wget 1.13.4 D'Oh!
    – speld_rwong
    Jun 20, 2016 at 13:19
0

What I would do, pragmatic approach ahead:

wget ....
rename 's/\.html\?.*/.html/' *.html*

This the Perl's rename command

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .