2

When rsyncing directory src to dest, is there any way I can make dest contain the last k sub-directories, in lexicographical order, of directory src, as in a moving window where sub-directories having fallen outside of the window are automatically removed from dest when re-running the same command at a later time?

For example, if k = 2, src contains

dir_1
dir_2
dir_3
dir_4
dir_5

and dest is empty, after running rsync, dest should contain dir_4 and dir_5. When, at a later time src, contains

dir_1
dir_2
dir_3
dir_4
dir_5
dir_6

and rsync is run again (with the very same command line), I want dir_4 to be deleted from dest, replaced by dir_6.

2

Yes, this is possible.

A /bin/sh solution:

#!/bin/sh

# number of entries to sync
k=2

# generate file list (sets $1, $2, $3 etc., collectively known as $@)
set -- src/*

# shift off all but the last $k of these entries
shift "$(( $# - k ))"

# create include patterns ($entry is not actually used,
# we work on the 1st element and then add it to the end and shift)
for entry do
    set -- "$@" --include="${1##*/}/***"
    shift
done

# run rsync
rsync --verbose --archive --delete --delete-excluded "$@" --exclude='*' src/ dest/

This creates a list of files in $@ with set first (this list is sorted lexicographically, because globs do that), then removes all but the last $k of these. The loop converts each src/element into --include=element/***. This inclusion pattern will make rsync consider the named element and anything below it. These inclusion patterns are used on the rsync command line together with --exclude='*' which will exclude everything not explicitly included (first match matters).

The rsync run uses --delete which will delete anything in the included subdirectories on the destination that is not available in the source directory, and --delete-excluded will additionally delete everything that was excluded.

Run the script with sh -x to see what happens.

You could do this in bash with arrays as well, but the syntax is a bit messy.

  • Great answer, thanks. While k is quite small in my case, the src directory contains tens of thousands of subdirectories, and I wonder whether this solution scales reliably. I had little luck finding out how many tokens $@ can handle from the man pages. – gd1 Jun 17 '18 at 11:39
  • 1
    @gd1 There should not be an issue with many subdirectories. The script will just use a bit of memory to store the pathnames as strings. The only issue is if k is very large in which case the command line that runs rsync may be too long. If you do run into issues, we would have to look at other solutions. – Kusalananda Jun 17 '18 at 11:49
  • 1
    @gd1 In general though, I'd recommend to not have more than a thousand or so directory entries in any directory, if at all possible, as traversing huge directories is slow. Utilities that does caching often has two or three levels of subdirectories in their top-level cache directory, for speed. – Kusalananda Jun 17 '18 at 12:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.