2

I'm trying to download threads of a web forum which contains many images with wget. These images are hosted in a separate domain, so I used the -H flag to allow downloading them. The complete wget order I'm using is:

wget -p -H -erobots=off https://example.com/

And when I use it almost all downloads except the main images. This is an example of one of those threads I am trying to download.

What I'm doing wrong?

Improve this question
3
  • Are you getting any error messages? If you right-click on an image that is not getting downloaded, and either save the URL or open image in a new tab, can you successfully use wget specifying just the URL of that image?
    – Mark Stewart
    Jul 6, 2018 at 15:55
  • I don't get any error messages and if I specify just the URL of the image it downloads correctly.
    – Rick
    Jul 6, 2018 at 16:27
  • @Rick have you finally solved it?
    – yssup
    Jan 24, 2021 at 21:28

1 Answer 1

Reset to default
1

This is a site specific problem. Running

 wget -H -p -r --user-agent="Mozilla/5.0" -nd --accept jpg,png https://suzuki88.mforos.com/194412/11461305-brico-con-mi-nuevo-suzuki-en-proceso-de-transformacion-muchas-fotos/

gives me a list of robot.txt.* files with content:

User-agent: *
Disallow:

which basically implies that the website thinks you are a robot trying to patch into some security vulnerability of the site. And the rule is to deny * (all) robots access to any part of the site afaiu.

Improve this answer
1
  • 2
    That robots.txt doesn't deny access, but allows to visit any part of the website AFAIK. Wikipedia says this: "This example tells all robots that they can visit all files because the wildcard * stands for all robots and the Disallow directive has no value, meaning no pages are disallowed: User-agent: * Disallow:"
    – nxnev
    Jul 6, 2018 at 21:12

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .