17

If I try wget on a webpage, I am getting the page as html. Is it possible to retrieve only text of a file without associated html ? (This is required for me since some of the HTML pages contains c program is getting downloaded with html tags. I have to open it in browser and manually copy the text to make a .c file.)

  • 1
    Actually it is possible, but you need to write some function that will parse the code from the page and save it as .c. It is not hard, but it depends on page structure. If you provide a link may be somebody will help you with exact code. Otherwise sed or perl are your friends. – rush Jul 9 '12 at 8:28
25

wget will only retrieve the document. If the document is in HTML, what you want is the result of parsing the document.

You could, for example, use lynx -dump -nolist, if you have lynx around.

lynx is a lightweight, simple web browser, which has the -dump feature, used to output the result of the parsing process. -nolist avoids the list of links at the end, which will appear if the page has any hyperlinks.

As mentioned by @Thor, elinks can be used for this too, as it also has a -dump option (and has -no-references to omit the list of links). It may be especially useful if you walk across some site using -sigh- frames (MTFBWY).

Also, keep in mind that, unless the page is really just C code with HTML tags, you will need to check the result, just to make sure there's nothing more than C code there.

  • Or use elinks which has support for frames. Use -no-references to achieve the same effect as nolist for lynx. – Thor Jul 12 '12 at 23:01
  • @Thor, thanks, I updated the answer to mention that. – njsg Aug 8 '12 at 8:30
5

If you don't have these other tools installed, only wget, and the page has no formatting just plain text and links, e.g. source code or a list of files, you can strip the HTML using sed like this:

wget -qO- http://address/of/page/you/want/to/view/ | sed -e 's/<[^>]*>//g'

This uses wget to dump the source of the page to STDOUT and sed to strip any < > pairs and anything between them.

You can then redirect the output of the sed command to the file you want to create using > :

wget -qO- http://.../ | sed -e 's/<[^>]*>//g' > downloaded_file.txt

NB: You may find that it has extra whitespace in the file that you don't want (e.g. lines are indented a few columns)

It may be easiest to use your text editor to tidy the file up that (or a source formatter as you're downloading C source code).

If you need to do the same simple thing to every line of the file you could include a command to do that in the sed command (here stripping one leading space):

wget -qO- http://.../ | sed -e 's/<[^>]*>//g;s/^ //g' > downloaded_stripped_file.txt
3

just to add another tool. I prefer w3m, which is a lynx like console browser. You may want to check out whats already available on your system.

w3m -dump website.html

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.