I would like to recreate the directory structure from a remote system to a local system and copy a sample of the files in each directory (e.g. copy only 5 files from each directory). There may be many levels of nested directories and many files in each directory.

My question is similar to this one, but with the added difficulty of the source directory being on a remote system: Copy directory structure with random number of files

I have tried to modify the solution mentioned in the above question by adding ssh commands, but I get confused managing the ssh commands through multiple pipes and I don't believe the last step with cpio can copy from remote systems:

find "$src" -type d -links 2 -exec \
    sh -c 'find "$1" -type f -print0 | shuf -z -n "$2"' sh {} "$nfiles" \; | \
    cpio -padmv0  "$dest"

I have investigated whether the filter rules in rsync can do this, but I don't believe they can as the files don't have a specific naming pattern that can be used.

Please let me know if there's a straightforward way to do this.

Edit: Solutions involving perl or python scripts are welcome. Please let me know what the best tools for the job are.

  • This looks like a job for Perl. Trying too hard to do it with just shell + ssh + rsync + cpio etc. will just mean a lot more false starts and wasted effort. – q.undertow Jun 1 at 23:06
  • Thanks. I added a comment welcoming perl and python solutions. – Dan Jun 1 at 23:41

I ended up writing the following script in Python to solve the problem. It could probably be faster, but it works. Feedback is welcome.

from pathlib import Path

from paramiko import SSHClient
from scp import SCPClient

ssh = SSHClient()

scp = SCPClient(ssh.get_transport())
source_dir = '/remote_data'
local_dest_dir = Path('/local_data')
num_files = 10

command = f"find {source_dir} -type d -links 2"
stdin, stdout, stderr = ssh.exec_command(command)
dirs = [Path(d.strip('\n')) for d in stdout.readlines()]
for dir in dirs:
    relative_dir = str(dir).replace(source_dir, '')
    local_dir = Path(local_dest_dir) / Path(relative_dir)
    local_dir.mkdir(exist_ok=True, parents=True)
    command = f"find {source_dir} -type f | shuf -n {num_files}"
    stdin, stdout, stderr = ssh.exec_command(command)
    files = [Path(f.strip('\n')) for f in stdout.readlines()]
    for file in files:
        scp.get(str(file), local_dir / file.name)
| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.