3

wget is a great tool to make a quick snapshot of a small site. As far as I know (I really hope I just couldn't find it in wget --help), wget can follow well-known HTML URL attributes only, like <a href=..., <img src=... and so on. However, sometimes a particular site might use non-standard attributes that represent real URLs that do not look like URLs for wget. Let' say, if a site has a "static" gallery with zoomed images, a particular image page might have something like this:

<div zoomed_img="/gallery/image.jpg">
    <img src="/gallery/image_small.jpg"/>
</div>

Thus, wget ignores the zoomed_img attribute with /gallery/image.jpg. My wget command is:

wget --recursive \
    --domains domain \
    --no-parent \
    --page-requisites \
    --no-clobber \
    --html-extension \
    --convert-links \
    http://domain/gallery

Is it possible to make wget follow custom URL HTML attributes?

  • did you try the --follow-tags command ? – Thomas Jan 31 '16 at 9:03
  • @Thomas just checked it. Specifying the div tag explicitly as --follow-tags=a,div,img resulted in the same effect. – Lyubomyr Shaydariv Jan 31 '16 at 9:11
2

wget (at least 1.16.3) does not allow a user to specify custom attributes. There has been a suggestion to extend the the --follow-tags option with a syntax like --follow-tags=a/href, but nobody followed up on that.

If you do not mind applying a quick hack to wget to support your scenario, see this commit for the work needed to add custom tags or attributes.

Alternatively, you can write a post-processor yourself.

  • Thanks for interesting links. Yes, I've checked out the latest wget source code, and wget seems to have nothing related to custom attributes (my wget version is 1.17.1.11-b305). As far as I understand C code, html_url.c and wget, --follow-tags is an intersection filter against interesting_tags initially filled with known_tags (thus there could be no way to add new tags? -- probably that was a reason why the suggestion was not followed up). I'll try to add the div/zoomed_img pair myself. Regarding post-processor: unfortunately I was looking for a Windows-friendly solution too. – Lyubomyr Shaydariv Jan 31 '16 at 12:25
  • I would like to accept this answer, as it really reveals that wget is unable to fulfill the request. I had the same idea of patching wget, and I just added TAG_DIV, and registered it in known_tags and tag_url_attributes locally. The local version of wget mirrored all I needed. Just 3 lines of code is not a big issue. I wish I could know C enough to add the support for tag/attribute pairs myself. – Lyubomyr Shaydariv Feb 1 '16 at 9:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.