1

I have a remote machine with a large number of numbered directories, like so:

dir1 dir2 dir3 ... dir40

each of which contain several numbered files:

file1 file2 file3 ... file2530

I want to copy only a selected range of the files in each directory. Since the files' names are identical in each directory, I want to re-create the directory hierarchy on my local machine. But since I don't want every file, I can't just use scp -r to copy every file in the directory.

I can't set up an automated connection with ssh keys on the remote machine, so I would prefer a method that doesn't involve repeated calls to a remote copy command. The files are also big, so I don't want to just copy the whole thing over and delete the ones I don't want with rm and brace expansion.

How can I copy a set of files from a remote machine, along with those files' parent directories, while preserving the directory structure and without copying every file in those directories?

3
  • First things first, do the deleting stuff at a 2. run of the program, when you've already saved your files. If the saving fails it's very unlikely to recover the rm'ed files. Also the file range, can you specify it?
    – ADDB
    May 31, 2017 at 21:13
  • @ADDB: file-{01-32}.out
    – Dan
    May 31, 2017 at 21:14
  • @ADDB: Also, I don't want to rm the copied over files. I suppose I could cp -rl to another directory on the remote and then delete the unwanted hardlinks before I transfer it over.
    – Dan
    May 31, 2017 at 21:16

1 Answer 1

Reset to default
2

You could use rsync, which will do only one ssh to the remote, and provide it with either a complete list of files, or a list of glob patterns of files to copy or not copy. For example,

rsync -navR --exclude='*-[4-9]?.out' --exclude='*-3[3-9].out' --exclude='*-???*.out' myremote:'dir*' mylocaldir

This would exclude filenames like file-40.out with 2 characters from 40 to 99, and also file-33.out to 39, and also file-100.out or bigger. Run the command with the -n option as shown to collect the list of names that would be transferred, and if this is ok remove the option to actually do the copy.

Note, rsync does not support braces {} in its glob patterns. Alternative ways of specifying the files to copy depend on how exotic your exclusion pattern is, but a foolproof method is to use -n and no exclude patterns to get the complete list of names, then edit this list and provide it as an --files-from list of files. You would also need to remove the dir* from the remote destination:

rsync -av --files-from=list  myremote: mylocaldir

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .