I have a program that is running on multiple machines and which is writing the results to ~/data/machine_name/, each time the results file exceeds 1MB the program moves on and write to a new file. (All machines have the same directory architecture).

I want to do an automatic backup of these files in case one of the machines fail. Knowing that the most recent accessed file is the one being written by the program, and hence, I would copy all the old files except this one. The next time the backup job fires, the most recent file omitted in the previous backup will be included and so on..

Right now I am using only a simple scp command, which copies even the file being open by the remote machine.

Is there anyway to achieve this? a command line utility or a script for such a job?

  • Are your files opened for writing and continuously kept open until full or do they get open and closed a lot and written line by line?
    – Caleb
    Jul 27, 2011 at 19:50
  • As far as a file is not Full, it still opened till it gets full, once full it's closed
    – 0xFF
    Aug 2, 2011 at 8:04

1 Answer 1


You can use rsyncbackup utility with --exclude option:

/usr/bin/rsync -avz --exclude='$EXCLUDED_DIR_OR_FILE' -e ssh /your/base/folder \
user@host_or_ip:/target/machine/path >> /optional/log/file.log

You can determine $EXCLUDED_DIR_OR_FILE variable by finding last updated file. There are several ways to find last updated file in a given directory, one of them is using ls and awk utilities together:

ls -lrt | awk '{ f=$NF }; END{ print f }'

Please note that this command prints last modified file name and not he full path to the file.

  • Parsing the output of ls is to be avoided.
    – Caleb
    Jul 27, 2011 at 19:45
  • @Caleb I just gave it as an example hint. Jul 27, 2011 at 20:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.