0

Here is the structure of the folders I would like to consider:

/mnt/network/

is my network drive and

/local/

is my local folder. What I want is to synchronize the content of these two folders (which represents about 350 Gb of data). I am aware of two popular strategies: rsync and unison. It seems that unison can be slow for such a large amount of data so I consider only rsync.

Because I would like to synchronize both folders, I run two commands. One from /local/ :

rsync -avzut --progress --files-from=<(find . -mmin -1440 -type f) /local/. /mnt/network/

and one from /mnt/network/ :

rsync -avzut --progress --files-from=<(find . -mmin -1440 -type f) /mnt/network/. /local/

With these commands, I limitate the launch of rsync to files that were updated over the last 24 hours (= 1440 minutes). I have two issues with this strategy:

  1. If I run the first command (to update the content of /mnt/network/) twice, newer files are copied twice: (a) the first time the command is executed - which is what I want - but also (b) the second time the command is executed. I believe (b) is not normal since at this stage files on /local/ and on /mnt/network/ are identical (I checked both size and timestamp). Could this be related with the combination of rsync and find ? I thought of adding the option --size-only: rsync -avzut --size-only --progress --files-from=<(find . -mmin -1440 -type f) /local/. /mnt/network/ but it does not seem a robust way to handle the issue.

  2. my second issue is that the find command is very slow when executed on /mnt/network/ So I am looking for a way to ssh to the server, execute the find command on it and transfer the result of this command back to my local computer and then make the file transfer (this file transfer cannot be ordered from the server as I do not want the server to access directly my local disk). I simply have no idea on how to do this...

  • 1
    Using rsync with a network mounted drive throws away much of what makes rsync awesome. Why not rsync -avzut --progress /local/. user@network:/path/ (and rsync -avzut --progress user@network:/path/. /local/)? Particularly since you're considering ssh to the server anyway … – The Sidhekin Oct 9 '15 at 18:35
  • Why do you only want to synchronize files newer than 24 hours? Why not just synchronize everything? Either rsync looks at every file to check if it should be synced, or in your example find must look at every file. What are you gaining in your example? – jamesbtate Oct 9 '15 at 18:37
  • @The Sidhekin, indeed, I could do this but actually my network is mounted on my local computer... – Alain Oct 9 '15 at 18:39
  • @Puddingfox, it is supposed to be a script that will be executed daily, hence the 24 hour check. What I am gaining in this example is the use of rsync on a very limited number of files. "find" can identify modified files much faster than "rsync" – Alain Oct 9 '15 at 18:40
  • 1
    What makes find faster than rsync? As long as aren't using the checksum option, both programs should just be doing a stat() on every file. – jamesbtate Oct 9 '15 at 18:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.