This is a strange situation I find myself in. I think I will have to write my own script but just wanted to know if I can use some rsync or some other options to get the job done.
I have to pick up the new and changed files from a windows shared folder. Randomly people will be putting batches of logs in there manually. Logs themselves are rather small but will be in huge numbers. I have mounted this shared folder using cifs.
I want to rsync the contents of this mounted folder with destination folder on my linux drive.
The destination drive is the source of data for logstash. To save space logstash has been configured to delete the files in the destination folder after processing them.
To my knowledge, rsync in this case is going to copy back the all of the files next time it is called.
Is there an option which can tell rsync to copy only new or changed files and not copy those files which it has already once copied to destination.
I think it is impossible as rsync does not maintain any records by itself. Or is there some advanced configuration possible? Or some other tool is there in linux ecosystem?