How to force wget to convert *all* downloaded links to relative ones?

Question

For having an offline version of Oracles Java tutorials¹, where all links to the Java API shall point to the also downloaded local version of the Java API docs, I use the following wget command (look out if trying: gets over 25,000 files):

wget -mkp -nH -nv -E  --accept-regex ".*javase/8/docs/api.*|.*javase/tutorial.*" http://docs.oracle.com/javase/8/

Unfortunately only a part of the links are converted to point correctly to the local files. All files under javase/tutorial are fine, but all the files pointing from there to the API files which are downloaded to javase/8/docs/api do not point to this local files, but instead point still to the online version.

Any ideas what's wrong?

––––

¹ I know there is a downloadable version of the tutorials, but it's API links point only to the online Java API doc.

Answer 1

It appears you have found a bug in wget. I tried several iterations of wget commands trying nail down the why. My first thought was that the regex was making wget treat it as two disjointed sets, and not linking between them. I worked out a non-regex version to get, seemingly, the same files, converting the options to long form and making them discrete (--mirror is compound) and adding in a log file and keeping the originals:

wget --recursive --convert-links --backup-converted --page-requisites --no-host-directories --level=inf --adjust-extension --include /javase/tutorial,/javase/tutorialJWS,/javase/8/docs/api https://docs.oracle.com/javase/8/index.html 2>&1 | tee wget.log

Grepping the files for comparison I found the it appears every link not converted was an absolute link in the original source. All the originally relative links were kept relative (not really converted). The output from wget claims to have 'converted' the files in question, but only non-anchor tags seem to have been touched.

I used the same options, and equivalent --include parameters on a site I control, after setting some links to absolute, including images. And got the same results: only non-anchor tags with absolute addresses were converted, absolute anchors were not affected.

For the size of your project, the prospect of converting the links to relative seems daunting, even with a script. I came across a PerlMonk post Change Absolute to Relative links in HTML files with a perl script that is aimed at the problem you now face.

I haven't tested it, but it has an option for making backup copies as a safety net, so it could be worth a try.

Meantime, I guess you get the privilege of reporting a bug on wget. Just for reference my version is:

$ wget --version
GNU Wget 1.14 built on linux-gnu.

+digest +https +ipv6 +iri +large-file +nls +ntlm +opie +ssl/openssl 

Wgetrc: 
    /etc/wgetrc (system)
Locale: /usr/share/locale 
Compile: gcc -DHAVE_CONFIG_H -DSYSTEM_WGETRC="/etc/wgetrc" 
    -DLOCALEDIR="/usr/share/locale" -I. -I../lib -I../lib 
    -fmessage-length=0 -grecord-gcc-switches -O2 -Wall 
    -D_FORTIFY_SOURCE=2 -fstack-protector -funwind-tables 
    -fasynchronous-unwind-tables -g 
Link: gcc -fmessage-length=0 -grecord-gcc-switches -O2 -Wall 
    -D_FORTIFY_SOURCE=2 -fstack-protector -funwind-tables 
    -fasynchronous-unwind-tables -g -lproxy /usr/lib64/libssl.so 
    /usr/lib64/libcrypto.so /usr/lib64/libz.so -ldl -lz -lz -lidn 
    ftp-opie.o openssl.o http-ntlm.o ../lib/libgnu.a