The image is generated once every n minutes, and I want to get each of these to my local directory. I had no success in getting the png-image using the command:

wget -r -l1 --no-parent -A.png http://url.com/home/images/

since it is blocked by the site, and I cannot use

wget http://url.com/home/images/filename.png

since the name of the images changes from update to update.

The image is however linked in http://url.com/home/index.html and I could get the file name from there. What is a robust way to do this? I know the pattern to search for: it is the only image that is called from the directory /home/images/.

  • 1
    I am successful with your first cmd. Why weren't you? What is missing, what is wrong with it? – polym Jun 19 '14 at 17:14
  • @polym True, there is nothing wrong with the command, it is just that the website does not allow me to get the images that way. Post edited. – Figaro Jun 19 '14 at 17:31
  • Can you print the response? – polym Jun 19 '14 at 17:33
  • There is no error, but wget only finds *.html files that are telling me that the page is missing. However, the png-file can be accessed directly through the browser. – Figaro Jun 19 '14 at 17:39
  • 1
    Try my suggestion below. If it still doesn't work, maybe the server is detecting wget requests because of the request header. In this case, you should emulate a browser header. – polym Jun 19 '14 at 17:45

Two strategies:

  • index.html grep strategy

It only works, as long there is only one png reference in index.html:

wget http://url.com/home/images/index.html
LINK=$(cat index.html | grep -zPo 'href=.*.png"')
LINK=${LINK#href=\"}; LINK=${LINK%\"}
wget --no-parent "http://url.com/home/images/$LINK"
rm index.html
  • Browser emulation by modifying request headers

Examples how to do that are here, here and here.

  • Must try the emulation, too. Thank you. I did try your script, but I am a bit slow on the syntax, when the png is embedded as "<img src="images/img123456.png" width=600 height=600>", how do I extract its path? It is also not the only png. It is however the only image with that path (for now). – Figaro Jun 19 '14 at 17:59
  • 1
    If its src, then change all href to src and wget from http://url.com/home/$LINK on line 5 instead of http://url.com/home/images/$LINK. I used simple string manipulation, first i remove the href=" part, second the last ". – polym Jun 19 '14 at 18:44
  • Got it working. Added a | head -1 to the grep so I only got the first (=correct) png. Not the most robust solution, but it is working. – Figaro Jun 19 '14 at 19:30
  • One more thing. How do I get this to work with cron? */5 * * * * /home/user/wget_testbed.sh does not generate any results. – Figaro Jun 19 '14 at 20:26
  • Got it working, had to specify the correct destination folder to wget – Figaro Jun 20 '14 at 16:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.