2

I am rsyncing a file from src to dst across a network. I’m navigating the many rsync options on the Linux man page, and need some help please.

A process on the destination responds to "newly arrived" files coming from the source. E.g. suppose the source file has a modification time of two days ago, but rsync didn't transfer for it until five minutes ago. I can detect that newly-arrived file based on its recent timestamp.

Therefore, I need this desired timestamp behavior:

  • dst file modification timestamp should represent the time at which content was last added by rsync transfer.
  • dst file modification timestamp should not change when rsync compares src and dst without transferring new content, i.e. when dst already matches src.

What rsync options provide this desired behavior?

5
  • 2
    The standard behavior with the archive option -a is to copy the timestamp of the original file to its copy. Do you want different timestamps? In that case why? Or should both the original file and the copy have a timestamp indicating when it was transferred?
    – sudodus
    Mar 9, 2023 at 18:39
  • 2
    Are you copying from a local filesystem to another local filesystem, or are you copying across a network between two servers? This makes a significant difference to any accurate answer Mar 9, 2023 at 19:00
  • I am copying across a network between servers.
    – Tomasso
    Mar 9, 2023 at 19:04
  • @sudodus - a process on the destination responds to "newly arrived" files coming from the source. E.g. suppose the source file has a modification time of two days ago, but rsync didn't transfer for it until five minutes ago. I can detect that newly-arrived file based on its recent timestamp.
    – Tomasso
    Mar 9, 2023 at 19:09
  • @Tomasso, I think you can use stat or some similar tool to look not only at the modification time but also the change time, which will be updated at the transfer. It means that you can use the standard behavior of rsync -a. This works at least when using linux file systems.
    – sudodus
    Mar 9, 2023 at 19:28

2 Answers 2

2

You can't do that efficiently. rsync uses the timestamp as its primary means of shortcutting long transfers across a network.

If you really want this timestamp behaviour,

  1. destination file timestamp should represent the time at which content was last added by rsync transfer
  2. destination file timestamp should not change when rsync compares source and destination without transferring new content

then you will, at best, end up with full checksums of every file for each file transfer consideration. So if you have transferred 100 files and those 100 files exist on source and destination, then each time you run rsync all 100 files will be checksummed to see if they have changed. As you can imagine, this is exceedingly inefficient.

rsync --archive --no-times --checksum /source/path remotehost:/destination/path

You haven't explained your use-case, but the usual way to work with rsync is to transfer file modification times so that it can avoid transferring files that appear to have already been transferred:

rsync --times /source/path remotehost:/destination/path    # or --archive

As usual, --archive = -a, --times = -t, --no-times = --no-t, --checksum = -c

2

It sounds like you are coding up an incremental backups use case.

Maintain a file in the src directory (or at least on the src host).

$ date +%s >> backed_up.txt

Now mtime on that bookkeeping file always corresponds to when the most recent backup began. Use find . -newer backed_up.txt -type f to compare timestamps and produce a list of files we wish to transfer. Feed rsync that list, and use --archive or whatever to preserve their stamps in the dst directory.


The key here is that for the target files we don't attempt to change how timestamps are being stored or used. We have a separate thing to model, and we put that where it belongs, in a bookkeeping file that is off to the side. So find / rsync use that timestamp rather than trying to shoehorn some novel usage into existing filesystems.


Here is one way for dst host to detect "newly arrived" files. It is cheaper than repeated stat() calls for mtime, as it only examines filenames. It assumes unique filenames that won't soon be re-used / appended.

#! /bin/bash

mv recent.{txt,old}
find some/dir -print | sort  > recent.txt
NEW=$(comm -1 -3 recent.{old,txt})

Now you can for FILE in $NEW iterate, feed the recent filenames to xargs, whatever your processing step might entail.


This approach relies on the standard utility comm. It appears in GNU TextUtils.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .