Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. It only takes a minute to sign up.
Sign up to join this community
I'm running wget like this:
wget --mirror --adjust-extension --convert-links --no-cookies http://tshepang.net -o log-main
I get a bunch of these messages:
Last-modified header missing -- time-stamps turned off.
I suppose that means that pages keep getting re-downloaded, even though I have them locally.
NOTE: I want this so that I don't have to re-download existing files each time I run the command mirror.
Did you try adding the -c parameter?
Excerpt from wget manual:
-c --continue
Beginning with Wget 1.7, if you use -c on a non-empty file, and it turns out that the server does not support continued downloading, Wget will refuse to start the download from scratch, which would effectively ruin existing contents. If you really want the download to start from scratch, remove the file.
Also beginning with Wget 1.7, if you use -c on a file which is of equal size as the one on the server, Wget will refuse to download the file and print an explanatory message. The same happens when the file is smaller on the server than locally (presumably because it was changed on the server since your last download attempt)---because ''continuing'' is not meaningful, no download occurs.
On the other side of the coin, while using -c, any file that's bigger on the server than locally will be considered an incomplete download and only "(length(remote) - length(local))" bytes will be downloaded and tacked onto the end of the local file. This behavior can be desirable in certain cases---for instance, you can use wget -c to download just the new portion that's been appended to a data collection or log file.
To my knowledge it should skip files that are already downloaded and of the same size.
