1

I have two bash arrays, one of them (named toMove) contains paths to files and directories to move (not copy) somewhere else, and the other one (named exclude) contains paths to files and directories to exclude from moving.

toMove=(
    tree/subtree1
    tree/subtree2
    tree/subtree3/leaffile3
)
exclude=(
    tree/subtree1/leafdir1
    tree/subtree2/leaffile2
)

# move code ?

This is the test directory structure:

mkdir -p tree/{subtree1/leafdir1,subtree2/leafdir2,subtree3/leafdir3}
touch tree/{subtree1/leaffile1,subtree2/leaffile2,subtree3/leaffile3}
tree tree
    tree
    ├── subtree1
    │   ├── leafdir1
    │   └── leaffile1
    ├── subtree2
    │   ├── leafdir2
    │   └── leaffile2
    └── subtree3
        ├── leafdir3
        └── leaffile3

Desired outcome after moving:

tree tree
    tree
    ├── subtree1
    │   └── leafdir1
    ├── subtree2
    │   └── leaffile2
    └── subtree3
        └── leafdir3

tree destination
    destination
    ├── subtree1
    │   └── leaffile1
    ├── subtree2
    │   └── leafdir2
    └── subtree3
        └── leaffile3

In rsync, there is an --exclude-from= option, that (in combination with the --remove-source-files option) does exactly what I want, but unfortunately rsync copies files but I need to move them (for performance reasons), if they are on the same filesystem.

The solution I have in mind, is using find to obtain a list of all paths (including the contents of the dirs) in the toMove array, iterating over this list and filtering out all paths starting with the paths in the exclude array. Is this the right approach to the problem, or is there a simpler and/or more elegant way (possibly using some standard utilities) to solve this?

UPDATE:

As it turns out, this problem is not as trivial as one might think first looking at it and this question was poorly phrased -- it should have been about "filtering out paths from other paths, preserving unaffected trees as a whole".

I ultimately went with the solution below, although it does not preserve empty leafdirs and has the second drawback that it moves each file, when it would be sufficient, to move only a parent tree in some cases.

...
comm \
    -23 \
    <(find "${toMove[@]}"  ! -type d 2>/dev/null | sort -u || true) \
    <(find "${exclude[@]}" ! -type d 2>/dev/null | sort -u || true) |
        parallel -j "$(nproc)" -- moveFile {}
...

(moveFile is an exported bash function that handles the moving. parallel was used to speed up the (few) times when the script will not operate on one filesystem.)

Stéphane Chazelas' answer told me a lot I didn't know before and seems to be the way to go, if zsh is an option.

5
  • In your desired output, there's subtree1 for instance in both tree and destination, so there has to be some copying involved in any case. Jul 15, 2022 at 16:12
  • That's correct. Parent directories need to be created in some cases.
    – anick
    Jul 15, 2022 at 16:15
  • If destination is on a different filesystem, a move implies a copy+delete anyway. Jul 15, 2022 at 16:15
  • The script will almost always operate on the same filesystem.
    – anick
    Jul 15, 2022 at 16:16
  • 1
    The standard pax utility or cpio or GNU cp with -l can do some copies as hard links (for non-directory files) and a rename() is like a link()+unlink(). You can exclude stuff with pax or cpio, or you can remove the things on either side after the linking. Jul 15, 2022 at 16:18

1 Answer 1

1

If all of source and destination are on the same filesystem, then a move is a rename() which is essentially the same as a link()+unlink().

The standard (though not widespread) pax command, the (formerly standard) cpio command and the GNU implementation of cp with -al can copy directory structures as symlinks (not for the directories which still need to be created anew), so with zsh instead of bash and the GNU implementations of cpio (for its -0/--null) and rm (for its -d/--dir option), you could do:

#! /bin/zsh -
src=tree
dst=destination
toMove=(
  subtree1
  subtree2
  subtree3/leaffile3
)
exclude=(
  subtree1/leafdir1
  subtree2/leaffile2
)

set -o extendedglob
autoload zargs

src=$src:P
dst=$dst:P

cd -- $src || exit

allToMove=( $^toMove{,/**/*}~(${(~j[|])exclude})(|/*)(ND) )
print -rNC1 -- $allToMove |
  cpio --pass-through --null --link --make-directories -- $dst &&
  zargs -r -- ${(Oa)allToMove} -- rm -d --

We end up creating all target directories anew (and linking all files within) even those we could have just renamed, so that could still be improved but at least none of the data of any non-directory files is being copied.

The directories that are created via --make-directories because they're not included in the list to be moved (like the subtree3 in the example) will not have their metadata (ownership, permission, etc) copied from the source.

You'll see some warnings about directories that can't be removed because they're not empty.

Some of the zsh features used there include:

  • $var:P expands to the absolute canonical path of the file stored in $var as if using standard realpath() on it (needed for $dst as we cd into $src, so the relative destination path would no longer refer to the same file thereafter).
  • $^array/x does rc/fish-like expansion of the array whereby if $array contains A and B as elements for instance, that becomes Ax Bx instead of A Bx.
  • A{x,y} is csh-like brace expansion which similarly expands to Ax, Ay.
  • **/ matches any level of subdirectories.
  • In glob~pattern, ~ is the and-not / except extendedglob operator used here to apply the exclusions.
  • (${(~j[|])exclude})(|/*) is the exclusion pattern constructed by joining the elements of the array with | (treated as a glob operator and not a literal | thanks to the ~ parameter expansion flag), to which we append (|/*) to match the elements or any file within.
  • (ND) glob qualifiers to apply Nullglob and Dotglob to those globs so hidden files are includes and no error is generated if the globs don't match.
  • print -rNC1 prints its arguments raw on 1 Column, NUL-delimited.
  • ${(Oa)array} expands in reverse array Order, so the leaves are removed before the branches they're on.
  • zargs is there to avoid the argument list too long errors if the list of files to remove is too large.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .