0

This question is essentially because of my inexperience using rsync, if it is duplicated i will promptly delete it.

I have a /dir-scr local folder, being fed continuously with production data from sensors readings. Every 10 minutes approx. a file with timestamp filename yyyymmddhhmmss.dat is being created, filled from 0 to 40MB approx. and closed. After 10 minutes approx., another file is created.

I've performed several batch executions of the updates:

$ rsync -azvP /dir-src/* host-dst:dir-src/

Which successfully copies and replaces partially filled files, i.e. copies complete files with 40MB and the current file with, let's say, 25MB, and at the next update, replace the 25MB file with the completed version with 40MB.

Currently I make a manual move operation for not deleting mistakenly any file not already copied.

Now i want to use one of the rsync flags such as:

  • --exclude
  • --delete-excluded
  • --delete-after
  • --remove-source-files

but i am unsure of their operation.

Question is, how to use rsync for moving files, let's say, older than 10 minutes, and delete from /dir-src if and only if they are already on host-dst:dir-dst/.

1
  • 1
    If you are looking to remove files from source after they have been sync'd, you need the --remove-source-files option. The --delete options deal with removing files at the destination that do not exist at the source.
    – Deathgrip
    Jun 20, 2017 at 17:07

1 Answer 1

2

I believe using rsyncs built in deletion options is not the right way to solve this problem. If you do something like the following:

$ rsync -azvP --remove-source-files /dir-src/* host-dst:dir-src/

you could be left in the situation where rsync transfers a partially filled file and then deletes it. If the file gets recreated with the same name then rsync will overwrite the copy on the destination machine. Thus, you will lose the first half of the file.

Maybe you want to consider this instead. Let rsync do its transfer as normal, then you can set up a cronjob that runs every ten minutes and does something like this.

$ find /dir-src -type f -mmin +15 -exec rm -f {} +

This will find all files under the dir-src directory that are older than 15 minutes and delete them. We are assuming here that the maximum time that it takes for a file to be filled is 15 minutes. You may want to tweak this value to suit your needs. Also, while testing please replace -exec rm -f {} + with -print, this will ensure that find is only picking up the files you actually want to delete before performing any destructive operation.

In your comment you say you have an unreliable network connection and are worried that rsync may not complete successfully. One method to get around this could be the following after creating an auxiliary directory:

  1. Every 15 minutes, move all files older than 15 minutes from dir-src to dir-aux.
  2. Every 10 minutes, rsync files from dir-aux to host-dst:dir-src.
  3. Every 60 minutes, delete all files in dir-aux.

This will ensure that rsync has had a chance to run 5-6 times before files are deleted. Depending on your connection this is probably enough. You can then scale that 60 minutes up or down depending on results from your testing.

6
  • My major fear is when trying to automate this, it is completely likely to loss connection onto the dir-src and there should be checking if the rsync was or not was executed succesfully the last time, hence every run of a rm command will be risky. So i can safely suppose not 15 mins, but maybe 1 hour, or more, but passing the full control to rsync for all those files, so that the current streaming is never interfered. My second choice actually is doing an executable, passing the files, listing the files and comparing them. I am sure there is a better way than that...
    – Brethlosze
    Jun 20, 2017 at 19:44
  • Or perhaps moving locally the files to a dir-aux folder with mv, for these older than 15 mins and then apply rsync on that folder?
    – Brethlosze
    Jun 20, 2017 at 19:48
  • 1
    I would go for your last solution if in your system mv (rename function) is atomic.
    – marc
    Jun 20, 2017 at 19:59
  • 1
    If your question has been answered please feel free to accept the answer.
    – rlf
    Jun 20, 2017 at 20:11
  • I will check the mv command perhaps today
    – Brethlosze
    Jun 21, 2017 at 14:26

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .