4

When fetching recursively with wget from an FTP server, how can I download only files modified after a given date?

The use case is that I need to sync with several servers nightly, but I only need the data from a couple days back. The --mirror flag would suffice, except that syncing the rest of the data (even just once) would take forever.

2 Answers 2

2

wget can mirror once and then after this only get later items with the --timestamp flag. I'd suggest reading the info page for it, which has a good explanation, and checking if this gives you the behaviour you are looking for.

1
  • In fact, --mirror implies --timestamping. Sep 24, 2011 at 20:05
0

No, that's not supported by wget unless you have already synced once, whether through wget or not (in which case, see --timestamp).

Here are some workarounds you could try:

  • If files contain dates in the name or path, and not just on the file modification time, you may be able to use regexes/directories to exclude old files.
  • Use something other than wget. This seems like a good candidate for a dedicated script. You can still use wget for a file at a time. You might also check out wpull, which I haven't used but I understand is similar to wget with scripting to determine whether to download pages.
  • If you have an HTTP download option in addition to FTP, you may be able to send custom HTTP cache directive headers which only download the file if it's modified after a certain date.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .