wget: Follow custom URL attributes

Question

wget is a great tool to make a quick snapshot of a small site. As far as I know (I really hope I just couldn't find it in wget --help), wget can follow well-known HTML URL attributes only, like <a href=..., <img src=... and so on. However, sometimes a particular site might use non-standard attributes that represent real URLs that do not look like URLs for wget. Let' say, if a site has a "static" gallery with zoomed images, a particular image page might have something like this:

<div zoomed_img="/gallery/image.jpg">
    <img src="/gallery/image_small.jpg"/>
</div>

Thus, wget ignores the zoomed_img attribute with /gallery/image.jpg. My wget command is:

wget --recursive \
    --domains domain \
    --no-parent \
    --page-requisites \
    --no-clobber \
    --html-extension \
    --convert-links \
    http://domain/gallery

Is it possible to make wget follow custom URL HTML attributes?

Answer 1

wget (at least 1.16.3) does not allow a user to specify custom attributes. There has been a suggestion to extend the the --follow-tags option with a syntax like --follow-tags=a/href, but nobody followed up on that.

If you do not mind applying a quick hack to wget to support your scenario, see this commit for the work needed to add custom tags or attributes.

Alternatively, you can write a post-processor yourself.

Answer 2

As wget doesn't allow to follow data-src, my solution was to use a proxy to replace all the "data-src" by "src" on the fly.

Using https://mitmproxy.org/ for example, with the following script:

"""
Fix the lazy loader SRC.
"""
from mitmproxy import http

def response(flow: http.HTTPFlow) -> None:
    if flow.response and flow.response.content:
        flow.response.content = flow.response.content.replace(
            b"data-src", b"src"
        )

Running the proxy:

mitmdump -s src.py --ssl-insecure

Telling wget to use the proxy:

wget -e use_proxy=yes -e http_proxy=localhost:8080 -e https_proxy=localhost:8080 --mirror --convert-links --adjust-extension --page-requisites --no-parent --progress=dot --recursive --level=6 --reject-regex "(.*)\?(.*)" --no-check-certificate https://website.com/

The result is that wget will mirror the whole website by passing throug the local proxy, which is editing the HTML on the fly before wget can read it. So images url are discovered.