Tell me more ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.
up vote 2 down vote favorite

I'm running wget like this:

wget --mirror --adjust-extension --convert-links --no-cookies http://tshepang.net -o log-main

I get a bunch of these messages:

Last-modified header missing -- time-stamps turned off.

I suppose that means that pages keep getting re-downloaded, even though I have them locally.

NOTE: I want this so that I don't have to re-download existing files each time I run the command mirror.

share|improve this question
Are you the owner of the page (if it really is tshepang.net). If so, is the web server apache and do you have any control of its configuration? Or at least is .htaccess enabled with options override allowed? – forcefsck Mar 18 '11 at 14:32
@forcefsck: Nah, the site is ran by Posterous. They have an API for reading all posts, but it's a learning curve for me these web tech. – Tshepang Mar 18 '11 at 14:47

2 Answers

up vote 1 down vote accepted
+100

Did you try adding the -c parameter?

Excerpt from wget manual:

-c --continue

Beginning with Wget 1.7, if you use -c on a non-empty file, and it turns out that the server does not support continued downloading, Wget will refuse to start the download from scratch, which would effectively ruin existing contents. If you really want the download to start from scratch, remove the file.

Also beginning with Wget 1.7, if you use -c on a file which is of equal size as the one on the server, Wget will refuse to download the file and print an explanatory message. The same happens when the file is smaller on the server than locally (presumably because it was changed on the server since your last download attempt)---because ''continuing'' is not meaningful, no download occurs.

On the other side of the coin, while using -c, any file that's bigger on the server than locally will be considered an incomplete download and only "(length(remote) - length(local))" bytes will be downloaded and tacked onto the end of the local file. This behavior can be desirable in certain cases---for instance, you can use wget -c to download just the new portion that's been appended to a data collection or log file.

To my knowledge it should skip files that are already downloaded and of the same size.

share|improve this answer
up vote 1 down vote

That means that the web server does not provide last modification info. Many servers hide that info for static content to manipulate the browser's cache.

You have instructed wget to ask for that info with --timestamping flag (which is redundant, it is implicitly enabled with --mirror). If you don't want wget to re-download the same files on one run, try this (untested)

wget -r -l inf -nc --adjust-extension --convert-links --no-cookies http://tshepang.net -o log-main

It isn't a good way to update an already existing mirror though (it won't re-download the same files even if they're changed), but AFAIK, there is no other workaround for wget.

edit: removed -N that accidentally left in the command line

share|improve this answer
I updated my question to exclude --timestamping flag. – Tshepang Mar 14 '11 at 12:09
Also, look at my question for clarification... I do want new files. – Tshepang Mar 14 '11 at 12:09
Unless you have access to the web server's config and can re-enable last-modification header, wget has no other way of knowing that the remote file is newer than the local one. – forcefsck Mar 14 '11 at 12:49
This won't be reliable, but maybe it can compare file sizes or, better still, the timestamp :) – Tshepang Mar 14 '11 at 13:45
2  
But that's your problem, timestamp is the missing Last-modified header and wget doesn't have a size-only check (it would also be unreliable anyway) – forcefsck Mar 14 '11 at 13:59
show 1 more comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.