Selectively copy from a collection of remote directories

Question

I have a remote machine with a large number of numbered directories, like so:

dir1 dir2 dir3 ... dir40

each of which contain several numbered files:

file1 file2 file3 ... file2530

I want to copy only a selected range of the files in each directory. Since the files' names are identical in each directory, I want to re-create the directory hierarchy on my local machine. But since I don't want every file, I can't just use scp -r to copy every file in the directory.

I can't set up an automated connection with ssh keys on the remote machine, so I would prefer a method that doesn't involve repeated calls to a remote copy command. The files are also big, so I don't want to just copy the whole thing over and delete the ones I don't want with rm and brace expansion.

How can I copy a set of files from a remote machine, along with those files' parent directories, while preserving the directory structure and without copying every file in those directories?

Answer 1

You could use rsync, which will do only one ssh to the remote, and provide it with either a complete list of files, or a list of glob patterns of files to copy or not copy. For example,

rsync -navR --exclude='*-[4-9]?.out' --exclude='*-3[3-9].out' --exclude='*-???*.out' myremote:'dir*' mylocaldir

This would exclude filenames like file-40.out with 2 characters from 40 to 99, and also file-33.out to 39, and also file-100.out or bigger. Run the command with the -n option as shown to collect the list of names that would be transferred, and if this is ok remove the option to actually do the copy.

Note, rsync does not support braces {} in its glob patterns. Alternative ways of specifying the files to copy depend on how exotic your exclusion pattern is, but a foolproof method is to use -n and no exclude patterns to get the complete list of names, then edit this list and provide it as an --files-from list of files. You would also need to remove the dir* from the remote destination:

rsync -av --files-from=list  myremote: mylocaldir