Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I use wget's built-in spider mode as a convenience sometimes to quickly check a local site for broken links. This morning I turned its attention to a production site that we'd just put major changes on, and it's coming up with 3 broken links, but it seems impossible to tell where they are! (It only says what they're linking to and there's no straightforward way of relating that alone back to a page.)

The options I'm currently using are wget -r -nv --spider http://www.domain.com/ -o /path/to/log.txt. Does anyone know of an option I'm overlooking, a way to read the output, or even a simple substitute for this command that will also let me know what file the links appear in (and ideally a line #)?

share|improve this question

You should be able to watch the web server logs, in conjunction with the wget run. Look for the 404's in the log file and pull the referrer field. That will tell you the page that contains the broken link.

It should then just be a matter of examining that page for the offending link.

share|improve this answer
Good idea. I forgot I asked this on here, actually! What I ended up doing was using it in combination with grep on my local copy of the site (particularly using the -n option to get line numbers). – user19866 Jul 10 '12 at 19:52

A good way (not involving the webserver logs) is to use the --debug flag and grep for ^Referer:

On the command line:

wget -r -nv --spider http://www.domain.com/ 2>&1 | egrep -A 1 '(^---response end---$|^--[0-9]{4}-[0-9]{2}-[0-9]{2}|^[0-9]{4}-[0-9]{2}-[0-9]{2} ERROR|^Referer:|^Remote file does not)'

You can do similar grepping on your log. Caveat: some wget are not compiled with the support for --debug

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.