0

I have not been able to recursively download some pages on a website that uses links within frames, and further links in those frames. It is from zhongwen.com, an online chinese etymology dictionary.

The html for a typical page looks like this:

<FRAMESET BORDER=1 FRAMEBORDER=1 ROWS=40,*>
    <FRAME ALIGN=LEFT SRC=http://zhongwen.com/main.htm SCROLLING=no MARGINHEIGHT=0 FRAMEBORDER=no NAME=mainFrame>
    <FRAMESET BORDER=1 FRAMEBORDER=1 COLS=250,*>
      <FRAME frameborder=no NORESIZE SRC=http://zhongwen.com/s/bushou.htm NAME=f1>
      <FRAME NORESIZE SRC=http://zhongwen.com/d/198/x126.htm NAME=f2>  
    </FRAMESET>
</FRAMESET>

Within the frame, the main link is to http://zhongwen.com/d/198/x126.htm, which has html:

<head>
<FRAMESET border=1 framespacing=0 frameborder=1 COLS="245,*">
<FRAME SRC="**../../d/198/d126.htm**" NAME="f3">
<!--
<FRAME MARGINHEIGHT=0 SRC="../../t/115.htm#23" NAME="f4">
-->
<FRAME SRC="../../d/198/t126.htm" NAME="f4"></FRAMESET>
</head>
</MAP></MAP>

There are several further links, of which one, http://zhongwen.com/d/198/x126.htm, has html:

<head><base target="f2"></head>
<BODY bgcolor="FFFFFF">
<A NAME="23"></A>
<IMG border=0 src="http://zhongwen.com/d/198/d126.gif" USEMAP="#a">
...

I have tried using wget and httrack, but they both download only the initial page. Answers to this question asked before haven't helped, nor have options on the man pages.

httrack -Y --near -%P -*p3 -r9999 -B http://zhongwen.com/cgi-bin/zipu.cgi?b5=%A5%5F

wget --restrict-file-names=nocontrol --ignore-length --html-extension --tries=3 --timeout=30 --no-http-keep-alive --cookies=off --page-requisites --convert-links -m -H --follow-tags=FRAME,IMG http://zhongwen.com/cgi-bin/zipu.cgi?b5=%A5%5F

Can anyone suggest anything else to try, or if I have gotten anything wrong?

Thank you

Improve this question

1 Answer 1

Reset to default
0

I tried with wget2, and noticed that robot.txt has, along other stuff:

Disallow: /s/
Disallow: /d/

You should respect the domain owner's rules and turn this verification off only for very good reasons. If the path was not disallowed, wget(the robot) would have retrieved over it.

Improve this answer
3
  • Thanks, I couldn't at all figure out what was going on. I hadn't come across wget2 before - thank you for that also.
    – Luke Sharkey
    Dec 3, 2020 at 14:09
  • Would you know a good reference that also has an explanation of /s/ and /d/? I'm struggling to find find one. Thanks.
    – Luke Sharkey
    Dec 3, 2020 at 14:17
  • @LukeSharkey /d/ is the disallowed path prefix, assuming the website url before /d/ and anything after such as zhongwen.com /d/ 198/x126.htm matches /d/
    – MARco
    Dec 3, 2020 at 14:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.