Copy directory structure and sample of files from remote system

Question

I would like to recreate the directory structure from a remote system to a local system and copy a sample of the files in each directory (e.g. copy only 5 files from each directory). There may be many levels of nested directories and many files in each directory.

My question is similar to this one, but with the added difficulty of the source directory being on a remote system: Copy directory structure with random number of files

I have tried to modify the solution mentioned in the above question by adding ssh commands, but I get confused managing the ssh commands through multiple pipes and I don't believe the last step with cpio can copy from remote systems:

find "$src" -type d -links 2 -exec \
    sh -c 'find "$1" -type f -print0 | shuf -z -n "$2"' sh {} "$nfiles" \; | \
    cpio -padmv0  "$dest"

I have investigated whether the filter rules in rsync can do this, but I don't believe they can as the files don't have a specific naming pattern that can be used.

Please let me know if there's a straightforward way to do this.

Edit: Solutions involving perl or python scripts are welcome. Please let me know what the best tools for the job are.

Answer 1

I ended up writing the following script in Python to solve the problem. It could probably be faster, but it works. Feedback is welcome.

from pathlib import Path

from paramiko import SSHClient
from scp import SCPClient

ssh = SSHClient()
ssh.load_system_host_keys()
ssh.connect('example.com')

scp = SCPClient(ssh.get_transport())
source_dir = '/remote_data'
local_dest_dir = Path('/local_data')
num_files = 10

command = f"find {source_dir} -type d -links 2"
stdin, stdout, stderr = ssh.exec_command(command)
dirs = [Path(d.strip('\n')) for d in stdout.readlines()]
for dir in dirs:
    relative_dir = str(dir).replace(source_dir, '')
    local_dir = Path(local_dest_dir) / Path(relative_dir)
    local_dir.mkdir(exist_ok=True, parents=True)
    command = f"find {source_dir} -type f | shuf -n {num_files}"
    stdin, stdout, stderr = ssh.exec_command(command)
    files = [Path(f.strip('\n')) for f in stdout.readlines()]
    for file in files:
        scp.get(str(file), local_dir / file.name)