Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have tried using wget --mirror http://tshepang.net/, but it only retrieves one page, "tshepang.net/index.html". Is this a bug in wget?

Here's the output, from using the --debug option:

DEBUG output created by Wget 1.12 on linux-gnu.

Enqueuing http://tshepang.net/ at depth 0
Queue count 1, maxcount 1.
[IRI Enqueuing `http://tshepang.net/' with None
Dequeuing http://tshepang.net/ at depth 0
Queue count 0, maxcount 1.
--2011-01-15 12:32:51--  http://tshepang.net/
Resolving tshepang.net... 66.216.125.32
Caching tshepang.net => 66.216.125.32
Connecting to tshepang.net|66.216.125.32|:80... connected.
Created socket 4.
Releasing 0x089e2be0 (new refcount 1).

---request begin---
GET / HTTP/1.0

User-Agent: Wget/1.12 (linux-gnu)

Accept: */*

Host: tshepang.net

Connection: Keep-Alive



---request end---
HTTP request sent, awaiting response... 
---response begin---
HTTP/1.1 302 Found

Server: nginx/0.7.65

Date: Sat, 15 Jan 2011 10:33:45 GMT

Content-Type: text/html; charset=utf-8

Connection: keep-alive

Status: 302 Found

Location: http://posterous.com/sso/verify/2d35d71b1e728dc99f3c153eaf6f8fa0?jumpto=%2F

X-Runtime: 3

Set-Cookie: cookies_enabled=true; path=/

Cache-Control: no-cache

Content-Length: 141

X-Varnish: 419207385

Age: 0

Via: 1.1 varnish

X-Cache: MISS



---response end---
302 Found

Stored cookie tshepang.net -1 (ANY) / <session> <insecure> [expiry none] cookies_enabled true
Registered socket 4 for persistent reuse.
Location: http://posterous.com/sso/verify/2d35d71b1e728dc99f3c153eaf6f8fa0?jumpto=%2F [following]
Skipping 141 bytes of body: [<html><body>You are being <a href="http://posterous.com/sso/verify/2d35d71b1e728dc99f3c153eaf6f8fa0?jumpto=%2F">redirected</a>.</body></html>] done.
--2011-01-15 12:32:52--  http://posterous.com/sso/verify/2d35d71b1e728dc99f3c153eaf6f8fa0?jumpto=%2F
conaddr is: 66.216.125.32
Resolving posterous.com... 184.106.20.99
Caching posterous.com => 184.106.20.99
Releasing 0x089e3e20 (new refcount 1).
Found posterous.com in host_name_addresses_map (0x89e3e20)
Connecting to posterous.com|184.106.20.99|:80... connected.
Created socket 5.
Releasing 0x089e3e20 (new refcount 1).

---request begin---
GET /sso/verify/2d35d71b1e728dc99f3c153eaf6f8fa0?jumpto=%2F HTTP/1.0

User-Agent: Wget/1.12 (linux-gnu)

Accept: */*

Host: posterous.com

Connection: Keep-Alive



---request end---
HTTP request sent, awaiting response... 
---response begin---
HTTP/1.1 302 Found

Server: nginx/0.7.65

Date: Sat, 15 Jan 2011 10:33:46 GMT

Content-Type: text/html; charset=utf-8

Connection: close

Status: 302 Found

Location: http://tshepang.net/sso/recovery/2d35d71b1e728dc99f3c153eaf6f8fa0?jumpto=%2F

X-Runtime: 7

Set-Cookie: _sharebymail_session_id=296a636c8ed3cb6e4e7cabb10256008a; domain=.posterous.com; path=/; HttpOnly

Cache-Control: no-cache

Content-Length: 142

X-Varnish: 2019529137

Age: 0

Via: 1.1 varnish

X-Cache: MISS



---response end---
302 Found
cdm: 1 2
Stored cookie posterous.com -1 (ANY) / <session> <insecure> [expiry none] _sharebymail_session_id 296a636c8ed3cb6e4e7cabb10256008a
Location: http://tshepang.net/sso/recovery/2d35d71b1e728dc99f3c153eaf6f8fa0?jumpto=%2F [following]
Closed fd 5
--2011-01-15 12:32:53--  http://tshepang.net/sso/recovery/2d35d71b1e728dc99f3c153eaf6f8fa0?jumpto=%2F
Reusing existing connection to tshepang.net:80.
Reusing fd 4.

---request begin---
GET /sso/recovery/2d35d71b1e728dc99f3c153eaf6f8fa0?jumpto=%2F HTTP/1.0

User-Agent: Wget/1.12 (linux-gnu)

Accept: */*

Host: tshepang.net

Connection: Keep-Alive

Cookie: cookies_enabled=true



---request end---
HTTP request sent, awaiting response... 
---response begin---
HTTP/1.1 302 Found

Server: nginx/0.7.65

Date: Sat, 15 Jan 2011 10:33:46 GMT

Content-Type: text/html; charset=utf-8

Connection: keep-alive

Status: 302 Found

Location: http://tshepang.net/

X-Runtime: 5

Set-Cookie: _sharebymail_session_id=cab0227db8c38f17e572984ee188dc5e; domain=tshepang.net; path=/; HttpOnly

Cache-Control: no-cache

Content-Length: 86

X-Varnish: 419207606

Age: 0

Via: 1.1 varnish

X-Cache: MISS



---response end---
302 Found
cdm: 1 2
Stored cookie tshepang.net -1 (ANY) / <session> <insecure> [expiry none] _sharebymail_session_id cab0227db8c38f17e572984ee188dc5e
Location: http://tshepang.net/ [following]
Skipping 86 bytes of body: [<html><body>You are being <a href="http://tshepang.net/">redirected</a>.</body></html>] done.
--2011-01-15 12:32:54--  http://tshepang.net/
Reusing existing connection to tshepang.net:80.
Reusing fd 4.

---request begin---
GET / HTTP/1.0

User-Agent: Wget/1.12 (linux-gnu)

Accept: */*

Host: tshepang.net

Connection: Keep-Alive

Cookie: _sharebymail_session_id=cab0227db8c38f17e572984ee188dc5e; cookies_enabled=true



---request end---
HTTP request sent, awaiting response... 
---response begin---
HTTP/1.1 200 OK

Server: nginx/0.7.65

Date: Sat, 15 Jan 2011 10:33:49 GMT

Content-Type: text/html; charset=utf-8

Connection: keep-alive

Status: 200 OK

ETag: "6ec7aeb4e15e3a80e733f7c2b5e00d6f"

X-Runtime: 1680

Cache-Control: private, max-age=0, must-revalidate

Content-Length: 66513

X-Varnish: 419207692

Age: 0

Via: 1.1 varnish

X-Cache: MISS



---response end---
200 OK
Length: 66513 (65K) [text/html]
Saving to: `tshepang.net/index.html'

     0K .......... .......... .......... .......... .......... 76% 25.7K 1s
    50K .......... ....                                       100% 39.3K=2.3s

2011-01-15 12:32:58 (27.9 KB/s) - `tshepang.net/index.html' saved [66513/66513]

Deciding whether to enqueue "http://tshepang.net/".
Already on the black list.
Decided NOT to load it.
Redirection "http://tshepang.net/" failed the test.
FINISHED --2011-01-15 12:32:58--
Downloaded: 1 files, 65K in 2.3s (27.9 KB/s)
share|improve this question
    
Well.. there's only one page in example.com :) –  wag Jan 15 '11 at 9:57
    
This isn't a bug, I can recursively download other pages. Run wget -d -r http://tshepang.net to see debugging results, maybe your site blocks all those requests or does not accept the wget user agent. –  wag Jan 15 '11 at 10:22
1  
It seems like all the redirection caused wget to interrupt the request. Try with --no-cookies. –  wag Jan 15 '11 at 11:49
    
@wag pls help me clean this up –  Tshepang Jan 15 '11 at 15:16
    
I don't think it's helpful to remove comments, it's hard to follow the flow of activities that lead to the solution (and now it look like I'm schizophrenic). –  wag Jan 15 '11 at 16:54

3 Answers 3

Assuming wget is in your path (if not, you’ll have to enter the full path) issue the following commands:

mkdir wget_files
cd wget_files
wget --mirror –-wait=2 --page-requisites --html-extension –-convert-links –-directory-prefix wget_files/example1 http://www.yourdomain.com
share|improve this answer
    
that doesn't work; I still get the same problem –  Tshepang Jan 15 '11 at 10:46
    
Why are you mentioning a windows path on a site about unix? –  Mark Loeser Jan 15 '11 at 18:47
    
@mark it's a mess, but I gotta admit it was helpful though: --mirror and --html-extension, and --convert-links all basically ended in my final command. –  Tshepang Jan 15 '11 at 20:35
up vote 5 down vote accepted

The --no-cookies option helped (thanks to wag):

It seems like all the redirection caused wget to interrupt the request. Try with --no-cookies.

This was determined from reading the attached log.

share|improve this answer

You also need to set -r for recursive and -l X for link depth, where X is an integer. It's also a good idea to set -A to set the list of acceptable file types to keep (otherwise you only get HTML files).

share|improve this answer
    
Actually, the manpage says --mirror implies -r and -l inf (unlimited depth). Am not so sure about -A, but it in my mirror I got images, javascript, and stylesheets, and all that without using that option. –  Tshepang Jan 17 '11 at 9:14
    
Ok, suit yourself, but I mirror sites all the time with wget -mrl 100 -A html,jpg,css. Maybe your copy of wget is broken then? Or maybe you should try it before you -1 somebody? –  bahamat Jan 17 '11 at 16:29
    
I did not mean to offend. What version of wget are you using? I use 1.12 and that's what I get. If u use the same, try without the -r and -A and see what you get. –  Tshepang Jan 17 '11 at 17:48

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.