# routing wget --recursive to stdout (as in -O -) rather than a file

I want to "scan" a website content, recursively, and pipe it to another program.

When I run:

wget -r -O - $SITE  I get WARNING: combining -O with -r or -p will mean that all downloaded content will be placed in the single file you specified.  Then, "download" process starts, but hangs after the first page is done. By "hangs" I mean, nothing seems to be going on, nor do I get back the command prompt until I hit Ctrl+C. I would prefer avoiding the option outputting wget to a temporary folder and scan file by file, as well as writing my own scrapper which seems to be the most redundant act one can think of at the end of 2012. - 1: why are you using the -O argument ? it is intended to output the results to a file. 2: I would like you will explain more on what your intentions are, since it is hard for me to understand & help you (i hope). – Hanan N. Dec 25 '11 at 10:31 from the wget man page: Similarly, using ‘-r’ or ‘-p’ with ‘-O’ may not work as you expect: Wget won't just download the first file to file and then download the rest to their normal names: all downloaded content will be placed in file. This was disabled in version 1.11, but has been reinstated (with a warning) in 1.11.2, as there are some cases where this behavior can actually have some use.  – Hanan N. Dec 25 '11 at 10:32 -O - is the way to get results routed to stdout, rather than to a file on disk, and this is what I am doing here. – Tzury Bar Yochay Dec 25 '11 at 11:44 try adding the -q . – Hanan N. Dec 25 '11 at 12:09 ## 3 Answers wget -r -O -$SITE | scanningProgram


will work as long as you don't care about filenames, dirnames, paths, etc

Here's an example

wget -r -O - http://ginatrapani.googlepages.com | sed 's/./X/g'


all outout from this site is converted to X chars

-
You are right, it does if you either provide a file name e.g. -O /tmp/single-file-for-entire-site or chain a program as in your sed example. however, if you simply do -O - which means routing to stdout, than it hangs after the first page, that seems like a bug in the gnu-wget. Thanks anyway. –  Tzury Bar Yochay Dec 26 '11 at 4:18
@TzuryBarYochay Does wget -q -O /dev/stdout work? –  Gilles Dec 26 '11 at 23:32

For anybody finding this topic by searching, as I did (after a year and a half), I found that curl routes to stdout by default, whereas wget doesn't seem to have been intended for that use. So my solution is: