3

I'm trying to download a website recursively using wget:

wget --recursive --page-requisites --no-parent --convert-links -e robots=off --domains orteil.dashnet.org orteil.dashnet.org/cookieclicker

It only downloads certain files, while leaving others. Upon closer inspection I figured out it will not download files that return 304 Not Modified.

This file returns 304 Not Modified in Inspector and is not downloaded. A file returning 304

This file returns 200 OK in Inspector and is downloaded. A file returning 200

I'm guessing this is because wget assumes it doesn't need to download files with 304. How can I force it to download the files?

1 Answer 1

5

304 Not Modified is sent by the server in response to a If-Modified-Since header. It is supposed to tell the client that the copy he has is up to date. This header in turn is sent by wget because the file already exists in the output directory with some timestamp.

So the obvious way to force a download is to start with an empty output directory.

You might be having issues because you are using --convert-links and not --backup-converted, since --convert-links changes the timestamp on the file (thus making it appear more recent).

Edit: in more recent wgets than I was testing with, one can also add --no-if-modified-since

0

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .