Tell me more ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.
up vote 3 down vote favorite

When recursively wget-ting a fairly large site, after downloading 18 MB or so, wget starts complaining "Unable to resolve host" and/or "Temporary failure in name resolution", even though I can look up the host just fine with host or dig.

On the advice of various fora, I tried:

  1. commenting out the hosts line in /etc/nsswitch.conf
  2. replacing potentially dodgy nameservers in /etc/resolv.conf with Google's 8.8.8.8 and 8.8.4.4
  3. explicitly adding the name to /etc/hosts

It doesn't seem to matter whether or not I set --no-dns-cache in the wget command line; the same thing happens.

I can't tell if wget is eventually going to finish, or if it's going to stay stuck in this state. It'd be nice if it finished, since I need it to --convert-links.

I'm running wget 1.11 (compiled from source, because 1.12 and later have broken the behavior of --no-clobber with --convert-links, at least for my purposes) on a Linux Mint 14 system.

What causes this? Is there a way to fix it?

share|improve this question
You may run wget through strace in order to see what really happens. With a bit luck this can be seen in the output: strace -tt -f -o wget.strace wget ... That file is going to be big (few 100 K at least). Interesting is just the last part before the crash. – Hauke Laging Feb 27 at 1:14
Some servers don't like being hammered, and accept only so many requests from a particular IP address. That is what the --limit-rate= and --wait= flags are for. – vonbrand Feb 27 at 1:19
A server blocking requests would not exhibit as a host lookup failure. How many file descriptors does the wget process have when the errors start occurring? lsof -p $(pgrep wget) – mindthemonkey Feb 27 at 2:05
@mindthemonkey I think you might be on to something -- I left it to complete overnight, and came back to find the conversion process complaining "Too many open files". – David Moles Feb 27 at 16:51

1 Answer

up vote 2 down vote accepted

Unfortunately the true fix will most likely be using the latest version of wget as the issues have probably been fixed there. 1.11 is about 5 years old. There is a 1.11.4 too if you happen to be using the original 1.11.

Alternatively (and this is horrible.. and might not work depending on what is triggering the file descriptor usage):

  • Increase the file ulimit for the user running wget.
  • Work out a way to resume your wget nicely.
  • Monitor the fd usage via lsof or a simple ls /proc/$pid/fd
  • Restart your wget when it gets close to your ulimit of files.

Otherwise you would need to trace what is leaking the fd's in wget and patch it. There may be known file descriptor leak patches you could backport to 1.11. I can't seem to find a public version control for the wget source though.

share|improve this answer
I'm pretty sure I saw this in 1.13 before I noticed the noclobber problem, but I suspect you're right about the file descriptors. And yes, while you can get a tarball of the release source, GNU is unfortunately quite a lot behind the times in terms of public source control. – David Moles Feb 27 at 19:47
@mindthemonkey: curl and Perl's WWW::Mechanize could be an alternative as well. – 0xC0000022L Feb 27 at 19:51
@0xC0000022L curl would take a whole lot of scripting though as it doesn't have the recursive/mirror features. I've not used WWW::Mechanize, but it looks the same, a module you could write a recursive perl getter with? – mindthemonkey Feb 27 at 20:31
@mindthemonkey: yes, you do the parsing yourself to resolve links, that's correct. – 0xC0000022L Feb 27 at 20:47

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.