Sync same directories between two linux servers

Question

There are two RHEL 7.2 linux servers located in different places. Both have same directory structure. Requirement is to keep certain directories of both servers in sync, i.e any modification in server1 should get reflected in server2 and vise versa, but if some file gets deleted locally it should not get deleted from remote server also. If some files get modified while link between those two servers is down, it should be copied as soon as link gets established.To implement this following script is made :

#!/bin/bash
EVENTS="CREATE,MOVED_TO,MODIFY"
inotifywait -e "$EVENTS" -m -r --format '%w%f' --fromfile list.txt|
while read FILE; do
    echo $FILE
    returnvalue=1
    while [[ $returnvalue -ne 0  ]]
    do
        rsync -azr $FILE backupserver:/$FILE
        returnvalue=$?
        echo $returnvalue
        if [[ $returnvalue -ne 0 ]]
        then
            sleep 60
        fi
   done
done

list.txt contains all the directories to be monitored. This is script is running in both the servers.

Problem : Whenever some modification is made in server1, it detects and copies it to server2 with return code 0. But that is detected in server2 as new modification and it tries to copy back to server1, as that file is present in server1 rsync returns an error code 23, so programs gets stuck.

What is the best solution to this problem?

Note: We cannot use --ignore-existing option as same file exists in both places which needs to be updated if content gets changed.

Answer 1

** first thought this might be a rsync option issue ** I think differently now!

Answer: use a local lock to claim the right to make a copy

I would suggest a simple test, if you do not have one, of one directory, - and first - just use the defaults, and see where that does not work as desired.

** deleted rsync stuff **

OK - backup 4 steps and look again.

Maybe the problem is the loop logic:

• ServerA has a change (to a LARGEFILE), it starts to copy it.
• Now, on ServerB, LARGEFILE is new/modified DURING the rsync/rcp copy, and tries to copy it back 'immediately'

I suspect this 'immediacy' lies at the root cause of problem.

My suggestion is to add a "LOCK" file of some type:

• place the lock locally
• check for lock remote, if found at remote, remove local lock and ignore local inotify
• if not found at remote, start copy of file, remove lock when copy finshes

Answer 2

You might replace your wait on MODIFY events by instead waiting on close_write ( file closed after being opened writeable), which will ensure you do not try to rsync files that are still changing.

Answer 3

The problem was due to temporary file creation as suggested in other answers. It got resolved when I added --temp-dir=/tmp as an rsync option.