How to make rsync dst timestamp reflect time at which content was last transferred from src?

Question

I am rsyncing a file from src to dst across a network. I’m navigating the many rsync options on the Linux man page, and need some help please.

A process on the destination responds to "newly arrived" files coming from the source. E.g. suppose the source file has a modification time of two days ago, but rsync didn't transfer for it until five minutes ago. I can detect that newly-arrived file based on its recent timestamp.

Therefore, I need this desired timestamp behavior:

• dst file modification timestamp should represent the time at which content was last added by rsync transfer.
• dst file modification timestamp should not change when rsync compares src and dst without transferring new content, i.e. when dst already matches src.

What rsync options provide this desired behavior?

Answer 1

You can't do that efficiently. rsync uses the timestamp as its primary means of shortcutting long transfers across a network.

If you really want this timestamp behaviour,

1. destination file timestamp should represent the time at which content was last added by rsync transfer
2. destination file timestamp should not change when rsync compares source and destination without transferring new content

then you will, at best, end up with full checksums of every file for each file transfer consideration. So if you have transferred 100 files and those 100 files exist on source and destination, then each time you run rsync all 100 files will be checksummed to see if they have changed. As you can imagine, this is exceedingly inefficient.

rsync --archive --no-times --checksum /source/path remotehost:/destination/path

You haven't explained your use-case, but the usual way to work with rsync is to transfer file modification times so that it can avoid transferring files that appear to have already been transferred:

rsync --times /source/path remotehost:/destination/path    # or --archive

As usual, --archive = -a, --times = -t, --no-times = --no-t, --checksum = -c

Answer 2

It sounds like you are coding up an incremental backups use case.

Maintain a file in the src directory (or at least on the src host).

$ date +%s >> backed_up.txt

Now mtime on that bookkeeping file always corresponds to when the most recent backup began. Use find . -newer backed_up.txt -type f to compare timestamps and produce a list of files we wish to transfer. Feed rsync that list, and use --archive or whatever to preserve their stamps in the dst directory.

---

The key here is that for the target files we don't attempt to change how timestamps are being stored or used. We have a separate thing to model, and we put that where it belongs, in a bookkeeping file that is off to the side. So find / rsync use that timestamp rather than trying to shoehorn some novel usage into existing filesystems.

---

Here is one way for dst host to detect "newly arrived" files. It is cheaper than repeated stat() calls for mtime, as it only examines filenames. It assumes unique filenames that won't soon be re-used / appended.

#! /bin/bash

mv recent.{txt,old}
find some/dir -print | sort  > recent.txt
NEW=$(comm -1 -3 recent.{old,txt})

Now you can for FILE in $NEW iterate, feed the recent filenames to xargs, whatever your processing step might entail.

---

This approach relies on the standard utility comm. It appears in GNU TextUtils.