Here is the structure of the folders I would like to consider:
is my network drive and
is my local folder. What I want is to synchronize the content of these two folders (which represents about 350 Gb of data). I am aware of two popular strategies: rsync and unison. It seems that unison can be slow for such a large amount of data so I consider only rsync.
Because I would like to synchronize both folders, I run two commands. One from /local/ :
rsync -avzut --progress --files-from=<(find . -mmin -1440 -type f) /local/. /mnt/network/
and one from /mnt/network/ :
rsync -avzut --progress --files-from=<(find . -mmin -1440 -type f) /mnt/network/. /local/
With these commands, I limitate the launch of rsync to files that were updated over the last 24 hours (= 1440 minutes). I have two issues with this strategy:
If I run the first command (to update the content of /mnt/network/) twice, newer files are copied twice: (a) the first time the command is executed - which is what I want - but also (b) the second time the command is executed. I believe (b) is not normal since at this stage files on /local/ and on /mnt/network/ are identical (I checked both size and timestamp). Could this be related with the combination of rsync and find ? I thought of adding the option --size-only:
rsync -avzut --size-only --progress --files-from=<(find . -mmin -1440 -type f) /local/. /mnt/network/ but it does not seem a robust way to handle the issue.
my second issue is that the find command is very slow when executed on /mnt/network/ So I am looking for a way to ssh to the server, execute the find command on it and transfer the result of this command back to my local computer and then make the file transfer (this file transfer cannot be ordered from the server as I do not want the server to access directly my local disk). I simply have no idea on how to do this...