lsyncd - syncing multiple client to same directory breaks sync

Question

Backgroud -

I'm trying to sync NAS with DR_NAS for HA purpose.
Unfortunately, there's a complicated problem with this.

1. NAS is on Alibaba cloud. They don't support NAStoNAS sync yet.
2. Best way I know is to install sync protocols between 2 NASs but I can't directly connect to NAS. Alibaba controls it.
3. I tried mounting them on a single system and use lsyncd/rsync between two NASs. But it failed.
4. The lead author of lsyncd says that lsyncd uses Inotify Wait, which uses Linux Kernel to monitor filesystem. But Linux Kernel doesn't watch NFS file system. It knows when the server itself writes file to NAS, but not when another system writes.
5. So I must setup lsyncd on every systems that writes files to NAS because that's the only way Inotify Wait knows.
6. Sync was set. If a system writes file in (PROD NAS):/data/A/, lsyncd detects it and report it to rsync server. Rsync server then writes file in the corresponding directory (DR NAS):/data/A/

---

Current State -

It seemed to work pretty well until one day I found out that the sync was broken.

It looks like the writers can't sync the same directory.

• When SystemA is writing on /data/A and SystemB is writing on /data/B, rsync server does it too.
• When SystemA and SystemB simultaneously write on /data/C, one of the sync breaks and only one system can write on /data/C from then.

Question

1. Can multiple lsyncd clients sync with the rsync sever and use the same directory?
2. Is there a better way to sync two remote NFS directories?

Answer 1

The problem is in "lsyncd uses Inotify Wait" - presumably at a the /data Directory level(?) which causes changes to the destination data to trigger the lsync program to be started.

If two sources are simultaneously updating the same target, you can't rely on those updates even being valid if they are updating the same file. There shouldn't be a problem with using the same directory.

If inotify was specified at a more granular level - such as file level, then provided only one source is writing to the file (and possibly other programs were reading from it) then there should be no problem.

So when just one of the SystemX is writing on /data/X there is no conflict.

However, when two of the SystemX are writing to /data/SAME you are going to miss some of the inotify directory level triggering events causing lsync to miss some updates.

An answer as to how you could sync the two NAS directories would require a detailed knowledge of your system - but a periodic sync where you run rsync (separately) against the two NAS systems could work - provided there are no duplicated files in the appropriate directory.

Answer 2

I found the reason why writing on the same directory could cause the sync disconnection.

Rsync Behavior:
Rsyncd is a daemon which is run by root user. Directory privileges are configured in rsyncd.conf. Sync will be broken if configured UID does not match target directory's UID even if it has all permissions.

---

My case: Files are stored in a date-rolling directories, like /data/2021/07/20, /data/2021/07/21 and so on.
Although systems never write the same files, they were destined to write under the same 'new' directory.

For example,
if one of lsyncd server detects there's a new file in "source:/data/2021/07/25/A" but directory "target:/data/2021/07/25/" doesn't exist yet, the rsync server will first create a directory "target:/data/2021/07/25" under root privileges, and then copy the file A. The privileges of directory "target:/data/2021/07/25" will not match with the source's until file "A" is fully copied. When file is transferred rsync will sync directory privileges.

The sync disconnection can happen because of this behavior. If source B detects "source:/data/2021/07/25" when source A is still writing under a directory that's yet root privilege, sync with source B and target will break.

I tried to workaround this behavior by creating daily directories in advance, but today one of the sync was broken after one of it froze after writing 0 byte file.