Copy the latest files from the directory

Question

I have the following files in a data directory: myfile1.txt myfile2.jnx myfile3.sur.

I use the below command to copy the contents of the above directory to another location:

scp data/* remotehost.example.com:/data/ 

However, I find an issue when the contents of the directory are dynamically changed after issuing the command. Like the file myfile3.sur changed to myfile3.sur1 by another application that works on this directory in the background while the first file is still copying.

I know shell would translate * to the all the file names in that directory when I trigger my command. Is there a way to tell linux to include the changed/updated files too?

Answer 1

While this is not the direct answer, I'd suggest using rsync for this: https://medium.com/@sethgoldin/a-gentle-introduction-to-rsync-a-free-powerful-tool-for-media-ingest-86761ca29c34

Answer 2

In order to ensure that the wanted file with name as is and content as is will be transfer make a script with the following logic:

mkdir data/temp.$$
cp data/* data/temp.$$
scp data/temp.$$/* remotehost.example.com:/data/ 
rm data/temp.$$

By copy-moving the files to a temp dir in the same server will ensoure that the later tansfer will get the file.

Note that if the files are constantly changed by background processes try to make dependancies with those processes.