1

I am doing corpus generation through curl or lynx of some web pages. I have written a program in bash to strip away the html code (e.g., when I do lynx -source myurl), but the problem is that some pages have JavaScript, which does not get stripped away.

Is there any way using curl or lynx to not get Javascript? I do want the html coding in the first instance to get my main body data, that's why I didn't use lynx -dump option (and instead used lynx -source).

I think I am asking for too much here. Or if you know about any JavaScript stripping app that can work in Cygwin environment, please let me know. Thanks for reading!

0

I don't see any mention of source-transformation in the manual page for curl. So I think the answer is "no" there.

For lynx, you get much the same answer: it treats <script> tags as almost the same as comments (with some special handling for the closing tag). It has done that since around 2000 (referring to the changelog).

But lynx does not (yet) have an option for stripping comments (and scripts) from the download.

You might be able to use tidy to reformat the file to simplify scripting a do-it-yourself script-removal tool. For example, using the -wrap with a large value (such as the file size) will put all of the HTML tags in the first column, allowing a simple script to parse the result and discard <script> through </script>, etc.

| improve this answer | |
  • Thanks; I think I will have to see if I can find some kind of pattern then, which is not present otherwise in the text and does not affect the main body text of the page, so that I can write a code to remove it. – greatbears Nov 19 '15 at 6:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.