Remote SSH copy files with filenames that contain certain strings

Question

I have two remote servers. One that I am currently connected to and one that I am trying to copy a lot of files to (10.10.0.13)

I have a series of files I need to copy within various directories of the format:

/opt/DR/output/1/a/csva1file.csv
/opt/DR/output/1/a/csva2file.csv
/opt/DR/output/1/b/csvb1file.csv
/opt/DR/output/1/b/csvb2file.csv
/opt/DR/output/1/b/csvb3file.csv
/opt/DR/output/1/b/csvb4file.csv
/opt/DR/output/1/c/csvc1file.csv
...
/opt/DR/output/30/a/csva1file.csv

And this continues for output/1 to output/40 folders. All the folders inside are identical and all the filenames inside will all contain similar strings, just with slight differences depending on the folder they are in.

I want to copy all the files that contain a1 from any directory to a folder in a remote server:

[email protected]:/data/landing/a/a1/

Similarly, I want to do this for all b1, c1, c2 etc. files and copy them to their respective places on the remote server.

I cannot seem to find a way to do this that doesn't involve writing multiple lines of code.

I have tried

cd /opt/DR/output/1/a/
scp -r -v *a1* [email protected]:/data/landing/a/a1/

which works but I want to copy all the *a1* csv files rather than having to do them one by one.

I have looked into globbing but don't think it can be used for my case. I have also looked at using paramiko/glob for python but I couldn't get that to work either. Ideally, I would like to do this with a bash shell script, but a python script would also work.

Hope this makes sense. Any help would be greatly appreciated. I can copy via SFTP or SCP.

Answer 1

It's simple bash script for your task. Just replace BASE string on your directory in local machine, and replace SSH_BASE string on your remote machine. I hope this helps. nesting is 2

 #!/usr/bin/env bash

BASE="/opt/DR/output"
SSH_BASE="/data/landing"
for i in `ls ${BASE}`
do
    if [ -d ${BASE}/${i} ]; then
        for j in `ls ${BASE}/${i}`
        do
            if [ -d ${BASE}/${i}/${j} ]; then 
             ssh [email protected] "mkdir -p ${SSH_BASE}/${i}/${j}"
             scp ${BASE}/${i}/${j}/*.* [email protected]:${SSH_BASE}/${i}/${j}/
            fi      
        done
    fi
done