14

I'm running wget like this:

wget --mirror --adjust-extension --convert-links --no-cookies http://tshepang.net -o log-main

I get a bunch of these messages:

Last-modified header missing -- time-stamps turned off.

I suppose that means that pages keep getting re-downloaded, even though I have them locally.

NOTE: I want this so that I don't have to re-download existing files each time I run the command mirror.

2
  • Are you the owner of the page (if it really is tshepang.net). If so, is the web server apache and do you have any control of its configuration? Or at least is .htaccess enabled with options override allowed?
    – forcefsck
    Mar 18, 2011 at 14:32
  • @forcefsck: Nah, the site is ran by Posterous. They have an API for reading all posts, but it's a learning curve for me these web tech.
    – tshepang
    Mar 18, 2011 at 14:47

1 Answer 1

11
+100

Did you try adding the -c parameter?

Excerpt from wget manual:

-c --continue

Beginning with Wget 1.7, if you use -c on a non-empty file, and it turns out that the server does not support continued downloading, Wget will refuse to start the download from scratch, which would effectively ruin existing contents. If you really want the download to start from scratch, remove the file.

Also beginning with Wget 1.7, if you use -c on a file which is of equal size as the one on the server, Wget will refuse to download the file and print an explanatory message. The same happens when the file is smaller on the server than locally (presumably because it was changed on the server since your last download attempt)---because ''continuing'' is not meaningful, no download occurs.

On the other side of the coin, while using -c, any file that's bigger on the server than locally will be considered an incomplete download and only "(length(remote) - length(local))" bytes will be downloaded and tacked onto the end of the local file. This behavior can be desirable in certain cases---for instance, you can use wget -c to download just the new portion that's been appended to a data collection or log file.

To my knowledge it should skip files that are already downloaded and of the same size.

2
  • What about if the file is smaller?
    – CJ7
    Feb 2, 2017 at 2:22
  • Yes but if there is on going download is in progress and when I run the same wget operation for that file they write into the same file so it will be corrupted
    – alper
    Aug 30, 2020 at 17:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.