I am working to get a data transfer service up to which I can copy only newly created files in /h/data/share/(company share) to /g/data/global/ (public dropbox), however the files in /g/data/global are being moved, which is causing me some issues because it keeps copying and syncing the same files over and over, which the other end is getting the many duplicates.

I have tried some of the following:

    cd /h/data/share/
       find . -newerct -1 > /h/data/sync.txt
       rsync -avzh --no-R --from-file=/h/data/sync.txt /g/data/global/

I keep getting too many arguments after running, this is supposed to be a service which can be auto run with a cronjob. Also, this will be moving thousands of .mp3, .jpg, .png, .doc, etc per day.

  • I am guessing the find command will find directories which have been changed (I have not seen -newerct before), and the rsync -a will cause all of these directories to be recursively copied, which may be rather more than you want. I suggest you add -type f to your find command. – icarus Nov 6 '19 at 8:21
  • What are the file systems behind it? rsync relies on change times and for e.g. FAT these are two-second intervals. Thus you would have to increase a time window there. see here – Fiximan Nov 6 '19 at 9:19
  • You mention Dropbox in the title, but there's no mentioning of it in the text. BTW, are you allowed to copy files from the company share to a public cloud storage (which the title implies you're trying to do)? – Kusalananda Nov 6 '19 at 10:33
  • Yes I am allowed, this is a file share we are sharing data with other entities. Only data that goes into this share will be shared out, however they want to pass it through the DMZ which is where my Linux box resides so it can handle the data passing. /h/data/share is mounted to a files ahre inside our firewalls, and /g/data/global/ is on the exportfs out to certain ips. – David Lashua Nov 6 '19 at 12:43
  • -newerct deals with created date rather than mt which is modified date. I think I might need to do a cp of some kind. – David Lashua Nov 6 '19 at 12:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.