Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have an ever growing remote directory of files that I'd like to sort and filter before I rsync them, with the goal of always keeping only the latest N files in my destination directory (i.e. a rotation scheme, more or less). Since rsync doesn't seem to have options for this, I've been using the technique of "inserting an arbitrary remote command", described here:

http://stackoverflow.com/q/950062/787842

with which I came up with this command, that I execute as a cron on the destination host:

rsync -vrzO --delete \
    -e ssh <remote_host>:'$(cd <remote_dir> && ls -t $PWD/* | head -n 25)' \
    <destination_dir>

This works well the first time (i.e. when the dest dir is empty): only the 25 freshest files get copied. But then when the window "slides forward" (i.e. when the arrival of a newer file should push an older one out, locally), the problem is that the --delete option doesn't work as I'd expect. My guess is that since the filtered out remote files still exist (i.e. they're just being temporarily hidden by the inserted '$(..)' command), then rsync simply cannot filter them out locally. Is my understanding correct, and is there a better way?

share|improve this question
add comment

2 Answers 2

up vote 3 down vote accepted

You can simulate your command as,

rsync -vrzO --delete -e ssh <remote_host>:'<remote_dir>/file1 <remote_dir>/file2' <destination_dir>/ i.e. substituting output of cd <remote_dir> && ls -t $PWD/* | head -n 2.

rsync's --delete works on directories and your command substitution is providing list of files. So --delete is not working.

Excerpt from man rsync:

--delete

This tells rsync to delete extraneous files from the receiving side (ones that aren’t on the sending side), but only for the directories that are being synchronized. You must have asked rsync to send the whole directory (e.g. "dir" or "dir/") without using a wildcard for the directory’s contents (e.g. "dir/*") since the wildcard is expanded by the shell and rsync thus gets a request to transfer individual files, not the files’ parent directory.

share|improve this answer
    
I'm not sure I understand the first part of our answer: what do you mean by "simulating my command"? So the conclusion is that what I want to do (with a single rsync call) is simply impossible, is that it? –  cjauvin Nov 25 '11 at 23:36
    
That means you can try your command by explicitly putting few file names in place of the command to be substituted which is giving you the list of latest files. –  Sachin Divekar Nov 26 '11 at 5:39
add comment

Mount the remote directory over sshfs. Then you can treat it as a local directory.

Now, to sort files by modification time, the easiest way is zsh's glob qualifiers. The 25 most recent files in a directory are

*(NDom[1,25])

So, if you want to copy first, then delete the extraneous files at the destination:

for x in source/*(NDom[1,25]); do
  y=destination/${x:t}
  if [[ ! -e $y || $x -nt $y ]]; then cp -p $x $y; fi
done
destination_files=(destination/*(NDom[1,25]))
if [[ ${#destination_files} -gt 25 ]]; then
  shift $((${#destination_files} - 25)) destination_files
  rm $destination_files
fi

Here's another method that deletes files before copying. This one deletes the oldest file when it's about to copy a new file and there is already a full complement at the destination.

destination_files=(destination/*(ND)); destination_count=$#destination_files
for x in source/*(NDom[1,25]); do
  y=destination/${x:t}
  if [[ $destination_count -gt 25 && ! -e $y ]]; then rm destination/*(NDom[-1]); fi
  if [[ ! -e $y || $x -nt $y ]]; then cp -p $x $y; fi
done

(Warning: the code above was typed directly into the browser, I did not test it.)

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.