2

First try with wget.

I'm trying to use it to match a regex and download all images starting with https://tse4.mm.bing.net/th?from a given url.

However, this doesn't seem to work.

The url in question is the following:

https://www.bing.com/images/search?q=kary+perry+the+one+that+got+away+cover&scope=images

(I just need to download a limited number of picture from bing)

I'm trying with this command here:

wget -nd -r -P save -A '^https://tse4.mm.bing.net/th?' https://www.bing.com/images/search?q=kary+perry+the+one+that+got+away+cover&scope=images

but it downloads the robot.txt files instead.

What am I doing wrong here? Thanks

1 Answer 1

3

The robots.txt file is there to stop you from performing bot operations on the site. wget will, by default abide by the wishes of the robots.txt file.

Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol.

It works likes this: a robot wants to vists a Web site URL, say http://www.example.com/welcome.html. Before it does so, it firsts checks for http://www.example.com/robots.txt, and finds:

User-agent: *
Disallow: /

The User-agent: * means this section applies to all robots. The Disallow: / tells the robot that it should not visit any pages on the site.


In bing's robots.txt file we can see the following:

User-agent: *
Disallow: /account/
Disallow: /bfp/search
Disallow: /bing-site-safety
Disallow: /blogs/search/
Disallow: /entities/search
...
Allow: /shopping/$
Allow: /shopping$
...
Disallow: /th?
Disallow: /th$
...

So they don't want you mass downloading from that path, you could try /shopping though.

If you find a way to bypass the robots.txt file and perform the operation you want, you will be acting maliciously and the site will likely ban your IP.


robotstxt.org


Conclusion

You likely aren't doing anything wrong (I'm not a wget expert by any means so there may be syntax errors as well), but the action is just not permitted.

1
  • Thanks for the explanation. Very clear. No i'm not gonna try to bypass it as it is not the important, i would just like to be able to automatically download a picture but it seems to be way to restricted (i tried bing as Google is even worse :) ) If you have any suggestions they are welcome, otherwise I'll just give up and search/download them manually when in need. Thanks again
    – Nick
    Jan 2, 2018 at 13:16

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .