3

When fetching recursively with wget from an FTP server, how can I download only files modified after a given date?

The use case is that I need to sync with several servers nightly, but I only need the data from a couple days back. The --mirror flag would suffice, except that syncing the rest of the data (even just once) would take forever.

2

wget can mirror once and then after this only get later items with the --timestamp flag. I'd suggest reading the info page for it, which has a good explanation, and checking if this gives you the behaviour you are looking for.

  • In fact, --mirror implies --timestamping. – John Flatness Sep 24 '11 at 20:05
0

No, that's not supported by wget unless you have already synced once, whether through wget or not (in which case, see --timestamp).

Here are some workarounds you could try:

  • If files contain dates in the name or path, and not just on the file modification time, you may be able to use regexes/directories to exclude old files.
  • Use something other than wget. This seems like a good candidate for a dedicated script. You can still use wget for a file at a time. You might also check out wpull, which I haven't used but I understand is similar to wget with scripting to determine whether to download pages.
  • If you have an HTTP download option in addition to FTP, you may be able to send custom HTTP cache directive headers which only download the file if it's modified after a certain date.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.