0

I have not been able to recursively download some pages on a website that uses links within frames, and further links in those frames. It is from zhongwen.com, an online chinese etymology dictionary.

The html for a typical page looks like this:

<FRAMESET BORDER=1 FRAMEBORDER=1 ROWS=40,*>
    <FRAME ALIGN=LEFT SRC=http://zhongwen.com/main.htm SCROLLING=no MARGINHEIGHT=0 FRAMEBORDER=no NAME=mainFrame>
    <FRAMESET BORDER=1 FRAMEBORDER=1 COLS=250,*>
      <FRAME frameborder=no NORESIZE SRC=http://zhongwen.com/s/bushou.htm NAME=f1>
      <FRAME NORESIZE SRC=http://zhongwen.com/d/198/x126.htm NAME=f2>  
    </FRAMESET>
</FRAMESET> 

Within the frame, the main link is to http://zhongwen.com/d/198/x126.htm, which has html:

<head>
<FRAMESET border=1 framespacing=0 frameborder=1 COLS="245,*">
<FRAME SRC="**../../d/198/d126.htm**" NAME="f3">
<!--
<FRAME MARGINHEIGHT=0 SRC="../../t/115.htm#23" NAME="f4">
-->
<FRAME SRC="../../d/198/t126.htm" NAME="f4"></FRAMESET>
</head>
</MAP></MAP>

There are several further links, of which one, http://zhongwen.com/d/198/x126.htm, has html:

<head><base target="f2"></head>
<BODY bgcolor="FFFFFF">
<A NAME="23"></A>
<IMG border=0 src="http://zhongwen.com/d/198/d126.gif" USEMAP="#a">
... 

I have tried using wget and httrack, but they both download only the initial page. Answers to this question asked before haven't helped, nor have options on the man pages.

httrack -Y --near -%P -*p3 -r9999 -B http://zhongwen.com/cgi-bin/zipu.cgi?b5=%A5%5F

wget --restrict-file-names=nocontrol --ignore-length --html-extension --tries=3 --timeout=30 --no-http-keep-alive --cookies=off --page-requisites --convert-links -m -H --follow-tags=FRAME,IMG http://zhongwen.com/cgi-bin/zipu.cgi?b5=%A5%5F

Can anyone suggest anything else to try, or if I have gotten anything wrong?

Thank you

1 Answer 1

0

I tried with wget2, and noticed that robot.txt has, along other stuff:

Disallow: /s/
Disallow: /d/

You should respect the domain owner's rules and turn this verification off only for very good reasons. If the path was not disallowed, wget(the robot) would have retrieved over it.

3
  • Thanks, I couldn't at all figure out what was going on. I hadn't come across wget2 before - thank you for that also. Dec 3, 2020 at 14:09
  • Would you know a good reference that also has an explanation of /s/ and /d/? I'm struggling to find find one. Thanks. Dec 3, 2020 at 14:17
  • @LukeSharkey /d/ is the disallowed path prefix, assuming the website url before /d/ and anything after such as zhongwen.com /d/ 198/x126.htm matches /d/
    – MARco
    Dec 3, 2020 at 14:51

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .