I am using wget --spider. It is consuming a lot of ram memory. How could I limit the use of ram memory from the wget command?

wget --spider -r -c http://www.example.org 2>&1 | grep '^--' | awk '{ print $3 }' | grep -v '\.\(css\|js\|png\|gif\|jpg\|JPG\)$' > example.txt

"-c" does not work with --spider? I would like to put the links in example.txt and be able to continue feeding the txt with the links at another time

  • What is it precisely that you're trying to do? Wget doesn't really do stateful recursive downloading. -c and -r do indeed work together, just not the way you're expecting it to – darnir Feb 12 at 13:04
  • @darnir Hello, I'm downloading a site that has more than 100 GB // 1 million pages. With this wget consumes practically all available ram memory (with or without --spider) – Rafael Feb 13 at 17:12
  • There are edge cases where recursive downloading can cause Wget's memory to shoot up massively. This is a known issue. I'd recommend you to post about this on bug-wget@gnu.org and we can help you to navigate your particular usecase – darnir Feb 13 at 17:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.