0

I have two remote servers. One that I am currently connected to and one that I am trying to copy a lot of files to (10.10.0.13)

I have a series of files I need to copy within various directories of the format:

/opt/DR/output/1/a/csva1file.csv
/opt/DR/output/1/a/csva2file.csv
/opt/DR/output/1/b/csvb1file.csv
/opt/DR/output/1/b/csvb2file.csv
/opt/DR/output/1/b/csvb3file.csv
/opt/DR/output/1/b/csvb4file.csv
/opt/DR/output/1/c/csvc1file.csv
...
/opt/DR/output/30/a/csva1file.csv

And this continues for output/1 to output/40 folders. All the folders inside are identical and all the filenames inside will all contain similar strings, just with slight differences depending on the folder they are in.

I want to copy all the files that contain a1 from any directory to a folder in a remote server:

[email protected]:/data/landing/a/a1/

Similarly, I want to do this for all b1, c1, c2 etc. files and copy them to their respective places on the remote server.

I cannot seem to find a way to do this that doesn't involve writing multiple lines of code.

I have tried

cd /opt/DR/output/1/a/
scp -r -v *a1* [email protected]:/data/landing/a/a1/

which works but I want to copy all the *a1* csv files rather than having to do them one by one.

I have looked into globbing but don't think it can be used for my case. I have also looked at using paramiko/glob for python but I couldn't get that to work either. Ideally, I would like to do this with a bash shell script, but a python script would also work.

Hope this makes sense. Any help would be greatly appreciated. I can copy via SFTP or SCP.

2
  • Do you have any list of those b1, c1, c2, etc?
    – DevilaN
    Feb 4, 2020 at 11:41
  • 1
    "for all b1, c1, c2 etc. files" I don't see the pattern. Is it that directory /1/a/ contains *a1* files, and /99/z/ would hypothetically contain *z99* files? If not, please clarify how we know what your "etc." is to represent. Feb 4, 2020 at 11:47

1 Answer 1

0

It's simple bash script for your task. Just replace BASE string on your directory in local machine, and replace SSH_BASE string on your remote machine. I hope this helps. nesting is 2

 #!/usr/bin/env bash

BASE="/opt/DR/output"
SSH_BASE="/data/landing"
for i in `ls ${BASE}`
do
    if [ -d ${BASE}/${i} ]; then
        for j in `ls ${BASE}/${i}`
        do
            if [ -d ${BASE}/${i}/${j} ]; then 
             ssh [email protected] "mkdir -p ${SSH_BASE}/${i}/${j}"
             scp ${BASE}/${i}/${j}/*.* [email protected]:${SSH_BASE}/${i}/${j}/
            fi      
        done
    fi
done
3
  • Please do not parse ls. Use at least ls -1 for more robustness. Better: for i in "$BASE"/*; do if the contents is not too much, otherwise while IFS= read -r i; do ...; done < <(ls -1 "$BASE"). Feb 4, 2020 at 14:10
  • When you'll run ssh command inside WHILE IFS, your script may stops. Not good idea. MB this: for I in "$BASE"/*, really better than my code, but no sure
    – spybull
    Feb 4, 2020 at 14:51
  • For the ssh command to work in a while-loop, stdin must be redirected from /dev/null. This could be achieved by explicilty redirecting with < /dev/null, or using the parameter -n to the ssh command, which does the same. Feb 4, 2020 at 15:30

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .