Continue a download if timestamp has not changed

Question

I'd like to mirror a site with large, updating database dumps. I don't entirely understand the -N and -c options and how they combine after reading the wget man page.

Is there a way to download a file, preferably with wget, so that:

• if the server's timestamp is newer than the local one, the download replaces the local file
• if the server's timestamp is the same as the local one, but the remote size is larger, wget continues the download

I happen to know the server supports sending modification times.

Answer 1

The wget manual has some more information than man wget:

A file is considered new if one of these two conditions are met:

1. A file of that name does not already exist locally.
2. A file of that name does exist, but the remote file was modified more recently than the local file.

To implement this, the program needs to be aware of the time of last modification of both local and remote files. We call this information the time-stamp of a file.

The time-stamping in GNU Wget is turned on using --timestamping (-N) option, or through timestamping = on directive in .wgetrc. With this option, for each file it intends to download, Wget will check whether a local file of the same name exists. If it does, and the remote file is older, Wget will not download it.

If the local file does not exist, or the sizes of the files do not match, Wget will download the remote file no matter what the time-stamps say.

In short, -N should do what you ask.