5

I am trying to sync content between two servers. The servers are identical builds (Ubuntu 12.04LTS) and have this cron job running to sync the content:

rsync -arzc --update --delete /htdocs/testing/www/cms_uploads/* root@123.123.123.123:/htdocs/testing/www/cms_uploads/

but I'm getting some strange results when I am testing the process as detailed below.

First test

  • Uploaded aaaa.jpg to Node2
  • syncs to Node1
  • Delete from Node1
  • Reappears on Node1
  • Delete from Node2
  • Removed from Node1

Second test

  • Uploaded bbbb.png to Node1
  • Syncs to Node2
  • Delete from Node2
  • Reappears on Node2
  • Delete from Node 1
  • Reappears on Node1

All I am trying to do is make sure the same content is exactly the same on both servers all the time. What am I doing wrong?

  • rsync checks modified time and then copy that file if modified time has updated – Rahul Patil Apr 30 '13 at 10:20
  • But the -a flag carries across the permissions, time etc – Craig Ward Apr 30 '13 at 10:28
  • Are you trying to mirror (which is unidirectional) or bidirectionally sync? – depquid Apr 30 '13 at 12:02
3

rsync is not setup to do two way syncs. Without specific help (e.g. sync from the machine that was changed) and a lot of luck, it cannot do so.

The luck is needed so that changes are infrequent and far apart. If both Node1 and Node2 get changed before the next sync is started (from either machine), some change does get lost on sync.

See also this

1

There's something fundamentally broken with your problem statement. Suppose that your servers are in synch, and then you create a file aaaa.jpg on Node 2. Should the next synchronization delete that spurious file from Node 2 (since it doesn't exist on Node 1, it must have been deleted), or should it copy the file to Node 1 (since it doesn't exist on Node 1, it must be newly created)?

The order in which you run the synchronizations will determine what happens in each case. This is practically guaranteed not to give the desired outcome in many cases. Worse, if the synchronizations run in parallel (so that one of the hosts is being updated by its own synchronization job and at the same time is being traversed by the synchronization job running on the other host), the result will look rather random.

Rsync is fundamentally designed for one-way synchronization. You can't just run two rsync jobs and hope to do two-way synchronization.

Unison is a file synchronizer designed for two-way synchronization. It's the closest thing to the right tool for your task. Set it up and run unison -auto remote.example.com://path/to/directory /path/to/directory on one of the hosts.

No matter what tool you use, there is the potential for conflicts, for example if the same file is replaced by two different versions on the two different machines. There is no good automated way to resolve such conflicts, so manual intervention will be required.

In most setups, the right thing is to designate a single server as the place to upload, and synchronize all the other servers from that master server. If someone uploads a file to a slave, make it relay the upload to the master; don't change anything locally. Whenever a file changes on the master, push it out to the slave(s).

1

rsync has a -u option that does:
"This forces rsync to skip any files which exist on the destination and have a modified time that is newer than the source file. (If an existing destination file has a modification time equal to the source file’s, it will be updated if the sizes are different.)"
So a shell script that contains
1) the command you gave plus the -u option and minus the -c option and
2) the same command but with the direction reversed
can sort-of accomplish a bi-directional sync but with two problems:
1) files deleted (or renamed) on the remote server will be copied over from the local b/c rsync will think it found a new file on the local, and
2) if a single file is changed on both locations between syncs, only the more recent changes will be preserved.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.