I have a computer periodically syncing folders of content with another computer using Resilio Sync. The receiving computer has a sorting process on an hourly cron, which analyses the folders and their contents before moving & cataloging them in a seperate filesystem.

My issue is the hourly cron will run and process folders without their full contents if the sync is incomplete. The hourly cron process requires the entire contents of the folder to process the contents correctly.

Is there a simple way of checking the contents of the receiving sync folder aren't open, i've looked into lsof, however perhaps there's an easier way?

I could switch from the resilio sync process to rsync if that would help.

  • A guaranteed-to-work and simple way is to have the system sending the data signal that it's done (note that the sender is the only one who knows when the data has been fully and successfully sent...). Can you just mv directories within the same filesystem from the remote system when the sync is done? Since a move within a filesystem is an atomic operation, it's one way to signal done. Or you can have the remote system create a softlink somewhere on the processing system, and that softlink is the signal. – Andrew Henle May 14 at 14:16
  • It seems simpler to run it from a single location. What if I were to run a single bash script on the receiving computer; Mount the remote using sshfs then: rsync remote1 tohere \ folder-organiser start \ rsync remote2 tohere \ folder-organiser start \ ...? – Kusuma Nassim May 14 at 22:47

I suspect there are many ways to do this. The first that came to me to to include a checksum. On the sending server you can run:

tar -cf - FILES | md5sum > my_sum.md5

Where we use tar to create (c) a file (f) onto stdin (-) from FILES which can be a glob, directory, or space delineated list of files which then gets piped to md5sum and the hash is saved in my_sum.md5.

On the receiving side you can add a check to your cron job to first try to find my_sum.md5 (if it's not there then obviously we haven't copied everything), if it is there, check to see that a similar generation of the checksum on the receiving side has a matching checksum.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.