1

This is a strange situation I find myself in. I think I will have to write my own script but just wanted to know if I can use some rsync or some other options to get the job done.

I have to pick up the new and changed files from a windows shared folder. Randomly people will be putting batches of logs in there manually. Logs themselves are rather small but will be in huge numbers. I have mounted this shared folder using cifs.

I want to rsync the contents of this mounted folder with destination folder on my linux drive.

The destination drive is the source of data for logstash. To save space logstash has been configured to delete the files in the destination folder after processing them.

To my knowledge, rsync in this case is going to copy back the all of the files next time it is called.

Is there an option which can tell rsync to copy only new or changed files and not copy those files which it has already once copied to destination.

I think it is impossible as rsync does not maintain any records by itself. Or is there some advanced configuration possible? Or some other tool is there in linux ecosystem?

Improve this question
2
  • 1
    Could you configure your rsync to move files rather than copy them, so that once it's copied the files from your source they will be deleted from there too?
    – roaima
    Nov 25, 2020 at 0:15
  • Oh I never thought that was possible. Thanks for hint. I will look it up.
    – NotAgain says Reinstate Monica
    Nov 25, 2020 at 0:33

1 Answer 1

Reset to default
1

If you're happy to have your rsync remove files from its source once it has successfully copied them, you can solve your requirement

rsync -av --remove-source-files ...

This is effectively a move rather than a copy.

Improve this answer
3
  • Will this remove files from nested directories? I see that although files were copied but nested directories still have the source files. Command: /usr/bin/rsync -av --remove-source-files --prune-empty-dirs --include "*/" --include="*.csv" --exclude="*" /Data/windowsSharedFolder /Data/tempLogsLocation I configured it to pick up only csv files and testing it out by copying csv files.
    – NotAgain says Reinstate Monica
    Dec 4, 2020 at 1:44
  • 1
    Seems ok. If it's not doing what you want then please update your question. (Incorporate the command and associated details in it as I if you'd included them from the start. Don't just write "edit" and put them on the end - it's not a forum.) If there's enough be information I'll see if I can address it in a revised answer
    – roaima
    Dec 4, 2020 at 8:07
  • 1
    Sorry for late response. I investigated it further. It turned out that somehow the permissions were revoked for the userid I was using. All good now.
    – NotAgain says Reinstate Monica
    Dec 14, 2020 at 0:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.