I'm trying to download threads of a web forum which contains many images with wget. These images are hosted in a separate domain, so I used the -H flag to allow downloading them. The complete wget order I'm using is:

wget -p -H -erobots=off https://example.com/

And when I use it almost all downloads except the main images. This is an example of one of those threads I am trying to download.

What I'm doing wrong?

  • Are you getting any error messages? If you right-click on an image that is not getting downloaded, and either save the URL or open image in a new tab, can you successfully use wget specifying just the URL of that image? – Mark Stewart Jul 6 '18 at 15:55
  • I don't get any error messages and if I specify just the URL of the image it downloads correctly. – Rick Jul 6 '18 at 16:27
  • @Rick have you finally solved it? – yssup Jan 24 at 21:28

This is a site specific problem. Running

 wget -H -p -r --user-agent="Mozilla/5.0" -nd --accept jpg,png https://suzuki88.mforos.com/194412/11461305-brico-con-mi-nuevo-suzuki-en-proceso-de-transformacion-muchas-fotos/

gives me a list of robot.txt.* files with content:

User-agent: *

which basically implies that the website thinks you are a robot trying to patch into some security vulnerability of the site. And the rule is to deny * (all) robots access to any part of the site afaiu.

  • 1
    That robots.txt doesn't deny access, but allows to visit any part of the website AFAIK. Wikipedia says this: "This example tells all robots that they can visit all files because the wildcard * stands for all robots and the Disallow directive has no value, meaning no pages are disallowed: User-agent: * Disallow:" – nxnev Jul 6 '18 at 21:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.