How to get text of a page using wget without html?

Question

If I try wget on a webpage, I am getting the page as html. Is it possible to retrieve only text of a file without associated html ? (This is required for me since some of the HTML pages contains c program is getting downloaded with html tags. I have to open it in browser and manually copy the text to make a .c file.)

Answer 1

wget will only retrieve the document. If the document is in HTML, what you want is the result of parsing the document.

You could, for example, use lynx -dump -nolist, if you have lynx around.

lynx is a lightweight, simple web browser, which has the -dump feature, used to output the result of the parsing process. -nolist avoids the list of links at the end, which will appear if the page has any hyperlinks.

As mentioned by @Thor, elinks can be used for this too, as it also has a -dump option (and has -no-references to omit the list of links). It may be especially useful if you walk across some site using -sigh- frames (MTFBWY).

Also, keep in mind that, unless the page is really just C code with HTML tags, you will need to check the result, just to make sure there's nothing more than C code there.

Answer 2

If you don't have these other tools installed, only wget, and the page has no formatting just plain text and links, e.g. source code or a list of files, you can strip the HTML using sed like this:

wget -qO- http://address/of/page/you/want/to/view/ | sed -e 's/<[^>]*>//g'

This uses wget to dump the source of the page to STDOUT and sed to strip any < > pairs and anything between them.

You can then redirect the output of the sed command to the file you want to create using > :

wget -qO- http://.../ | sed -e 's/<[^>]*>//g' > downloaded_file.txt

NB: You may find that it has extra whitespace in the file that you don't want (e.g. lines are indented a few columns)

It may be easiest to use your text editor to tidy the file up that (or a source formatter as you're downloading C source code).

If you need to do the same simple thing to every line of the file you could include a command to do that in the sed command (here stripping one leading space):

wget -qO- http://.../ | sed -e 's/<[^>]*>//g;s/^ //g' > downloaded_stripped_file.txt

Answer 3

just to add another tool. I prefer w3m, which is a lynx like console browser. You may want to check out whats already available on your system.

w3m -dump website.html