1

We are using rsync to backup one external drive to another external drive. When the two drives are connected to our Ubuntu machine, it works perfectly:

rsync -avzh --exclude='_quarantine'  /media/archive00/ /media/archiveX/archive00/

However, when we try to do the same operation on the same drive connected to a local server over ssh, it tries to copy all the files every time, not updating only those that have different update times:

rsync -avzh -e ssh --exclude='_quarantine' /media/archive00/ my_user@10.0.0.12:/media/archiveX/archive00/

The first 10 lines when I include the -i option:

sending incremental file list
.d..t...... BTEVC/
<f..t...... BTEVC/Untitled41.mov
<f..t...... BTEVC/UntitledX100.mov
<f..t...... BTEVC/gfxCardStatus-2.3.zip
<f..t...... BTEVC/BTEVCformBook/aud.ECPP.SCAmp/0d20180825203546p+13057989928.m4a
<f..t...... BTEVC/BTEVCformBook/aud.ECPP.SCAmp/0d20180826210032p+13057989928.m4a
<f..t...... BTEVC/BTEVCformBook/aud.ECPP.SCAmp/0d20180827174045p+18002662278.m4a
<f..t...... BTEVC/BTEVCformBook/aud.ECPP.SCAmp/0d20180904171935p+12123769623.m4a
<f..t...... BTEVC/BTEVCformBook/aud.ECPP.SCAmp/0d20180905190057p+18593229553.m4a
5
  • 1
    Is the target filesystem some sort of Windows filesystem that stores timestamps with imprecise precision, or does the target filesystem not preserve permission and/or user/group ownerships? Also, the default remote transport for rsync is ssh, so using -e ssh is not necessary. – Kusalananda Nov 15 '19 at 11:16
  • Definitely sounds like a owner/group issues. Try with options --usermap=*:<owner-id-according-to-target-FS> and --groupmap=*:<goup-owner-id-according-to-target-FS>. – SYN Nov 15 '19 at 11:18
  • @Kusalananda the other machine is debian, but the two drives are formatted to exFAT. @SYN hmm, it seems with the --usermap and --groupmap options it is still copying the files every time (but interestingly not the unlisted files, i.e. those with a '.' before them). Realized also that /etc/fstab was specifying that the drive on the server be mounted as root, and changed that also, but same results. – BTEVC Nov 15 '19 at 12:13
  • Add the -i flag and add the output to your question. It will show what criteria rsync thinks is responsible for the copy. – BowlOfRed Nov 15 '19 at 16:11
  • 1
    @BowlOfRed Ok, I added the first 10 lines of output. – BTEVC Nov 16 '19 at 2:44
1

From your output:

<f..t...... BTEVC/Untitled41.mov

The lowercase t indicates that rsync is (attempting to) set the time on the destination to match the source. Everything else after the f is blank indicating they match (so you don't have to look at user permissions being the cause).

You might want to investigate the times before and after a run to see how they differ.

Suggestions:

  • Try --inplace. See if anything changes
  • Copy a single file (so the output is smaller) and then up the verbosity. With -vvv you can see it try to set the transfer time similar to:

    [...]
    recv mapped dest.file of size 598
    got file_sum
    set modtime of .dest.file.5s3OoJ to (1573876681) Fri Nov 15 19:58:01 2019
    renaming .dest.file.5s3OoJ to dest.file
    [...]
    

See if anything odd is reported around that point (and that the actual timestamp matches afterward)

$ stat dest.file
  File: dest.file
  Size: 598             Blocks: 1          IO Block: 65536  regular file
Device: 8788005h/142114821d     Inode: 19140298416802240  Links: 1
Access: (0755/-rwxr-xr-x)  Uid: (197609/ compusr)   Gid: (197121/    None)
Access: 2019-11-15 20:26:49.597544100 -0800
Modify: 2019-11-15 19:58:01.309978600 -0800
Change: 2019-11-15 20:26:49.598968100 -0800
 Birth: 2019-11-15 20:26:49.594376800 -0800
4
  • Interesting, --inplace does not solve the problem, but your answer helped uncover a half-solution. Started reading about --inplace in this answer, and realized that --no-whole-file is assumed across a network. If --whole-file is used, it still copies the files that were there from before again, but at least it starts where it left off. – BTEVC Nov 16 '19 at 11:09
  • @BTEVC a big part of using rsync across a network is that it will only copy changed parts of changed files. Try to avoid forcing --whole-file because that removes the advantage. – roaima Nov 16 '19 at 11:14
  • Interestingly also, the behavior is the same as with --whole-file if only one file is transferred without that option. Will try to look at the details of the transfer to see if there's anything helpful in the output with the -vvv option tomorrow. – BTEVC Nov 16 '19 at 11:23
  • @roaima ok, right, good to know – BTEVC Nov 16 '19 at 11:25
1

One underlying problem is that "the two drives are formatted to exFAT" (from a comment).

The exFAT filesystem can only maintain timestamps to a two-second resolution, so you need to tell rsync this.

Fortuitously it has an option to manage this inaccuracy:

--modify-window When comparing two timestamps, rsync treats the timestamps as being equal if they differ by no more than the modify-window value. This is normally 0 (for an exact match), but you may find it useful to set this to a larger value in some situations. In particular, when transferring to or from an MS Windows FAT filesystem (which represents times with a 2-second resolution), --modify-window=1 is useful (allowing times to differ by up to 1 second).

Try rsync --modify-window=1 ....

1
  • --modify-window=1 doesn't seem to do it, but it is good to know for the future that exFAT has this behavior. – BTEVC Nov 16 '19 at 10:37
-1

What about using the "-u"-option? This will update only newer files.

The "-a"-option only sets the options below:

-a, --archive archive mode; equals -rlptgoD (no -H,-A,-X)
...
-r, recursive
-l, links
-p, perms
-t, times
-g, groups
-o, owner
-D, device specials

With the "-u"-option i never had any problems only copying new files. So it should be:

rsync -avzhu -e ssh --exclude='_quarantine' /media/archive00/ my_user@10.0.0.12:/media/archiveX/archive00
1
  • No -u doesn't do it either, unfortunately. – BTEVC Nov 15 '19 at 12:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.