7

wget is a great tool to make a quick snapshot of a small site. As far as I know (I really hope I just couldn't find it in wget --help), wget can follow well-known HTML URL attributes only, like <a href=..., <img src=... and so on. However, sometimes a particular site might use non-standard attributes that represent real URLs that do not look like URLs for wget. Let' say, if a site has a "static" gallery with zoomed images, a particular image page might have something like this:

<div zoomed_img="/gallery/image.jpg">
    <img src="/gallery/image_small.jpg"/>
</div>

Thus, wget ignores the zoomed_img attribute with /gallery/image.jpg. My wget command is:

wget --recursive \
    --domains domain \
    --no-parent \
    --page-requisites \
    --no-clobber \
    --html-extension \
    --convert-links \
    http://domain/gallery

Is it possible to make wget follow custom URL HTML attributes?

2
  • did you try the --follow-tags command ?
    – Thomas
    Commented Jan 31, 2016 at 9:03
  • @Thomas just checked it. Specifying the div tag explicitly as --follow-tags=a,div,img resulted in the same effect. Commented Jan 31, 2016 at 9:11

2 Answers 2

6

wget (at least 1.16.3) does not allow a user to specify custom attributes. There has been a suggestion to extend the the --follow-tags option with a syntax like --follow-tags=a/href, but nobody followed up on that.

If you do not mind applying a quick hack to wget to support your scenario, see this commit for the work needed to add custom tags or attributes.

Alternatively, you can write a post-processor yourself.

2
  • Thanks for interesting links. Yes, I've checked out the latest wget source code, and wget seems to have nothing related to custom attributes (my wget version is 1.17.1.11-b305). As far as I understand C code, html_url.c and wget, --follow-tags is an intersection filter against interesting_tags initially filled with known_tags (thus there could be no way to add new tags? -- probably that was a reason why the suggestion was not followed up). I'll try to add the div/zoomed_img pair myself. Regarding post-processor: unfortunately I was looking for a Windows-friendly solution too. Commented Jan 31, 2016 at 12:25
  • I would like to accept this answer, as it really reveals that wget is unable to fulfill the request. I had the same idea of patching wget, and I just added TAG_DIV, and registered it in known_tags and tag_url_attributes locally. The local version of wget mirrored all I needed. Just 3 lines of code is not a big issue. I wish I could know C enough to add the support for tag/attribute pairs myself. Commented Feb 1, 2016 at 9:48
1

As wget doesn't allow to follow data-src, my solution was to use a proxy to replace all the "data-src" by "src" on the fly.

Using https://mitmproxy.org/ for example, with the following script:

"""
Fix the lazy loader SRC.
"""
from mitmproxy import http

def response(flow: http.HTTPFlow) -> None:
    if flow.response and flow.response.content:
        flow.response.content = flow.response.content.replace(
            b"data-src", b"src"
        )

Running the proxy:

mitmdump -s src.py --ssl-insecure

Telling wget to use the proxy:

wget -e use_proxy=yes -e http_proxy=localhost:8080 -e https_proxy=localhost:8080 --mirror --convert-links --adjust-extension --page-requisites --no-parent --progress=dot --recursive --level=6 --reject-regex "(.*)\?(.*)" --no-check-certificate https://website.com/

The result is that wget will mirror the whole website by passing throug the local proxy, which is editing the HTML on the fly before wget can read it. So images url are discovered.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .