Tell me more ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.
up vote 1 down vote favorite

I want to "scan" a website content, recursively, and pipe it to another program.

When I run:

wget -r -O - $SITE

I get

WARNING: combining -O with -r or -p will mean that all downloaded content
will be placed in the single file you specified.

Then, "download" process starts, but hangs after the first page is done. By "hangs" I mean, nothing seems to be going on, nor do I get back the command prompt until I hit Ctrl+C.

I would prefer avoiding the option outputting wget to a temporary folder and scan file by file, as well as writing my own scrapper which seems to be the most redundant act one can think of at the end of 2012.

share|improve this question
1  
1: why are you using the -O argument ? it is intended to output the results to a file. 2: I would like you will explain more on what your intentions are, since it is hard for me to understand & help you (i hope). – Hanan N. Dec 25 '11 at 10:31
from the wget man page: Similarly, using ‘-r’ or ‘-p’ with ‘-O’ may not work as you expect: Wget won't just download the first file to file and then download the rest to their normal names: all downloaded content will be placed in file. This was disabled in version 1.11, but has been reinstated (with a warning) in 1.11.2, as there are some cases where this behavior can actually have some use. – Hanan N. Dec 25 '11 at 10:32
-O - is the way to get results routed to stdout, rather than to a file on disk, and this is what I am doing here. – Tzury Bar Yochay Dec 25 '11 at 11:44
try adding the -q . – Hanan N. Dec 25 '11 at 12:09

2 Answers

up vote 1 down vote 
wget -r -O - $SITE | scanningProgram

will work as long as you don't care about filenames, dirnames, paths, etc

Here's an example

wget -r -O - http://ginatrapani.googlepages.com | sed 's/./X/g'

all outout from this site is converted to X chars

share|improve this answer
You are right, it does if you either provide a file name e.g. -O /tmp/single-file-for-entire-site or chain a program as in your sed example. however, if you simply do -O - which means routing to stdout, than it hangs after the first page, that seems like a bug in the gnu-wget. Thanks anyway. – Tzury Bar Yochay Dec 26 '11 at 4:18
@TzuryBarYochay Does wget -q -O /dev/stdout work? – Gilles Dec 26 '11 at 23:32
up vote -1 down vote

using -r or -p with -O may not work as you expect: Wget won't just download the first file to file and then download the rest to their normal names: all downloaded content will be placed in file.try wget \ --recursive \ --page-requisites \ --html-extension \ --convert-links \ --restrict-file-names=windows \ --domains website.org \ --no-parent \ $SITE

share|improve this answer
my question was not how to download entire site with wget, I have no interest in downloading entire site and save it locally in disk, rather scan its content by an external program – Tzury Bar Yochay Dec 25 '11 at 11:42

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.