I am trying to crawl a website and want wget to recursively follow all the links (that are within the same domain) but only download the page if the URL of that page matches a pattern.

I tried something like this:

wget https://www.abc.def --no-clobber -r -e robots=off -U mozilla --domains abc.def --html-extension --accept-regex 'https://www.abc.def/g/h/.*'

It indeed only downloads pages with 'https://www.abc.def/g/h/.*' pattern. However, it doesn't try to follow other links at all.

Any idea?

Thanks in advance!

  • how about if you add --mirror? – DopeGhoti Jun 1 '17 at 16:20
  • what does it really do? I don't quite understand it in manpage: --mirror Turn on options suitable for mirroring. This option turns on recursion and time-stamping, sets infinite recursion depth and keeps FTP directory listings. It is currently equivalent to -r -N -l inf --no-remove-listing. – Hei Jun 2 '17 at 6:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.