1
vote
3answers
37 views

How can I use wget to create a list of URLs from an index.html?

So wget has an ability to recursively download files, however it does it one file at a time. I would like to pass in a directory URL, and for each URL it encounters in the recursion for it to spawn ...
3
votes
1answer
751 views

How to download all html files from a URL's directory with wget?

I'd like to use wget to download a tutorial in a website, but this tutorial is in a specific directory. Suppose the URL is the tutorial main page http://www.abc.com/aaa/bbb/ccc/index.html. But the ...
1
vote
1answer
447 views

wget recursive with files without extension

I'm not very good with Linux so I have maybe a basic question. I've search around for the "how to" and I couldn't find anything. I'm trying to get a site with wget. The problem is that it: Have ...
5
votes
3answers
4k views

wget - How to download recursively and only specific mime-types/extensions (i.e. text only)

How to download a full website, but ignoring all binary files. wget has this functionality using the -r flag but it downloads everything and some websites are just too much for a low-resources ...
2
votes
1answer
2k views

How do I create a local copy of a complete website section from OSX using wget?

This question follows from: How do I create a local copy of a complete website section from OSX using curl? After discovering OSX's native curl wouldn't do this task I downloaded wget from here: ...
1
vote
3answers
1k views

routing wget --recursive to stdout (as in -O -) rather than a file

I want to "scan" a website content, recursively, and pipe it to another program. When I run: wget -r -O - $SITE I get WARNING: combining -O with -r or -p will mean that all downloaded content ...