241

I am trying to set up rsync to synchronize my main web server to the remote server by adding newly generated file to the latter.

Here is the command that I use:

rsync -avh --update -e "ssh -i /path/to/thishost-rsync-key" remoteuser@remotehost:/foo/bar /foo/bar

But it seems that the web server actually transfers all files despite the '--update' flag. I have tried different flag combinations (e.g. omitting '-a' and using'-uv' instead) but none helped. How can I modify the rsync command to send out only newly added files?

Improve this question
4
  • 2
    If you want to sync files to the remote server, shouldn't it come last? i.e. rsync /foo/bar remoteuser@remotehost:/foo/bar
    – ostrokach
    Jun 28, 2015 at 13:07
  • You could be interested by this webpage. Also, be careful with the filesystem. If the server's filesystem doesn't support all that -a option implies (owner, group, perms, ...) it should be the cause of our issue.
    – ppr
    Jul 6, 2015 at 10:42
  • 2
    None of the answers solves the problem where the local files were not fully copied to the remote destination, and you only want to rsync newer files.
    – Axel Bregnsbo
    Nov 10, 2016 at 12:29
  • 3
    I just found the -c option. It skips existing files based on a checksum, instead of modification time or size.
    – haheute
    Mar 1, 2018 at 13:34

5 Answers 5

Reset to default
304

From man rsync:

--ignore-existing       skip updating files that exist on receiver

--update does something slightly different, which is probably why you are getting unexpected results (see man rsync):

This forces rsync to skip any files which exist on the destination and have a modified time that is newer than the source file. (If an existing destination file has a modification time equal to the source file's, it will be updated if the sizes are different.)

Improve this answer
8
  • 3
    Well actually I tried the command with '--ignore-existing' instead of '--update'. It finished fast but does not put the new files into remote host. Any ideas? Thanks
    – supermario
    Mar 12, 2013 at 2:34
  • 19
    Man pages suck (that's right I went there), case in point, does --ignore-existing mean don't transfer files which exist on receiver, or don't transfer files if some file/folder with the same name exists on the receiver?
    – puk
    Nov 20, 2013 at 9:05
  • 9
    --update does skip files when the mtimes are identical (which is not what the wording implies). I tested this. I believe the wording would be better understood if it said "only source files which are newer than destination will be copied".
    – Octopus
    Nov 28, 2015 at 7:41
  • 5
    @Octopus "only source files which are newer than destination will be copied" ...or source files that have same modification time as their destination file counterparts, but have different sizes. I think that is an important point. (Chris already covered that, thanks)
    – Jonathan Komar
    Apr 21, 2017 at 8:24
  • 1
    So you need --ignore-existing and -r (recursive). -v (verbose) is nice to have too :-)
    – DutchUncle
    Feb 4, 2018 at 18:57
68

The issue might be caused by different user/group IDs on source and target servers. My case was similar, having all files transferred instead of only the modified/new ones. The solution was to use parameters -t (instead of -a), and -P (equivalent to --partial --progress):

rsync -h -v -r -P -t <source> <target>

or shorter (thanks @Manngo);

rsync -hvrPt <source> <target>

This transfers only new files, and files already existing but modified. Parameter -a does too much, like user and group ID sync, which in my case can not work as I have different users and groups on my source and target systems. Therefore with -a all my source and target files were always regarded as "different".

The parameters in detail:

  • -h: human readable numbers
  • -v: verbose
  • -r: recurse into directories
  • -P: --partial (keep partially transferred files) +
            --progress (show progress during transfer)
  • -t: preserve modification times
Improve this answer
3
  • 5
    +1 for '-a does too much` (indeed -rlptgoD, so permission get not flattended, groups and owner preserved –not my typical play-safe backup...)
    – Frank N
    Oct 18, 2018 at 11:39
  • 2
    Worked for me. I prefer to type rsync -hvrPt for convenience.
    – Manngo
    Jan 19, 2020 at 6:01
  • @Manngo Thx for the suggestion!
    – t0r0X
    Jul 15, 2022 at 10:48
1

The problem you describe is probably because you are missing the trailing / on the source directory. As a result, rsync will copy all the files twice: the first time to /foo/bar and the second time to /foo/bar/bar. Thereafter it will efficiently copy updates to /foo/bar/bar.

The correct command should be this:

rsync -avh -e "ssh -i /path/to/thishost-rsync-key" remoteuser@remotehost:/foo/bar/ /foo/bar
Improve this answer
-1

From my experience with rsync, a 1TB partition copying is too large to be efficient. It takes rsync forever to process it. Instead, do it by subdirectories. That is, run rsync for each main subdirectory. It goes a lot faster if it doesn't have to juggle tens of thousands of files.

Improve this answer
2
  • 8
    Maybe you ran out of memory and your system started swapping?
    – Marcel Burkhard
    Feb 26, 2016 at 21:24
  • 3
    I regularly run rsync across TBs of data without problem (nine years later, mind)
    – roaima
    Jul 15, 2022 at 10:50
-1

Notice that rsync --ignore-existing -raz --progress /var/www/88021064/var [email protected]:/usr/home/xxx/public_html/var/ will not not do anything, the way it worked for me is rsync --ignore-existing -raz --progress /var/www/88021064**/var/*** [email protected]:/usr/home/xxx/public_html/var/, and notice that will not update hideen files for hideen files only execute one more time /var/www/88021064/var/.[^.]*

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .