I need to scp the files from machineB and machineC to machineA. I am running my below shell script from machineA. I have setup the ssh keys properly.

If the files are not there in machineB, then it should be there in machineC.


readonly PRIMARY=/data01/primary/.
readonly FILERS_LOCATION=(machineB machineC)
readonly MAPPED_LOCATION=/bat/data/snapshot
PARTITION=(0 3 5 7 9)


scp david@${FILERS_LOCATION[0]}:$dir1/weekly_1980_[$el]_200003_5.data $PRIMARY || scp david@${FILERS_LOCATION[1]}:$dir2/weekly_1980_[$el]_200003_5.data $PRIMARY

Now my question is if you take a look into my above scp command, I have [$el], I need to replace this with PARTITION, which means I need to iterate PARTITION and replace [$el] with each number in PARTITION.

So my scp command should look like this if I iterate PARTITION -

scp david@machineB:/bat/data/snapshot/20131222/weekly_1980_0_200003_5.data /data01/primary/. || scp david@machineC:/bat/data/snapshot/20131222/weekly_1980_0_200003_5.data /data01/primary/.

scp david@machineB:/bat/data/snapshot/20131222/weekly_1980_3_200003_5.data /data01/primary/. || scp david@machineC:/bat/data/snapshot/20131222/weekly_1980_3_200003_5.data /data01/primary/.

scp david@machineB:/bat/data/snapshot/20131222/weekly_1980_5_200003_5.data /data01/primary/. || scp david@machineC:/bat/data/snapshot/20131222/weekly_1980_5_200003_5.data /data01/primary/.

scp david@machineB:/bat/data/snapshot/20131222/weekly_1980_7_200003_5.data /data01/primary/. || scp david@machineC:/bat/data/snapshot/20131222/weekly_1980_7_200003_5.data /data01/primary/.

scp david@machineB:/bat/data/snapshot/20131222/weekly_1980_9_200003_5.data /data01/primary/. || scp david@machineC:/bat/data/snapshot/20131222/weekly_1980_9_200003_5.data /data01/primary/.
  1. How do I iterate PARTITION in such a way, so that I can make the above SCP command?
  2. And as you can see, I am copying the files one by one into machineA /data01/primary/ folder. Is there any better way of doing this? Meaning, can I copy all the files together in one shot instead of doing it one by one?

To copy all the partitions, use a wildcard:

scp david@machineB:/bat/data/snapshot/20131222/weekly_1980_*_200003_5.data /data01/primary/. || scp david@machineC:/bat/data/snapshot/20131222/weekly_1980_*_200003_5.data /data01/primary/.

If you to copy only partitions 0, 3, 5, 7 , 9, then use:

scp david@machineB:/bat/data/snapshot/20131222/weekly_1980_[03579]_200003_5.data /data01/primary/. || scp david@machineC:/bat/data/snapshot/20131222/weekly_1980_[03579]_200003_5.data /data01/primary/.

If the partition list is specified by a variable, the simplest would be:

scp david@machineB:/bat/data/snapshot/20131222/weekly_1980_[$PARTITION]_200003_5.data /data01/primary/. || scp david@machineC:/bat/data/snapshot/20131222/weekly_1980_[$PARTITION]_200003_5.data /data01/primary/.

If it is specified in an array:

PARTITION=(0 3 5 7 9)
scp david@machineB:/bat/data/snapshot/20131222/weekly_1980_["${PARTITION[*]}"]_200003_5.data /data01/primary/. || scp david@machineC:/bat/data/snapshot/20131222/weekly_1980_["${PARTITION[*]}"]_200003_5.data /data01/primary/.
  • Thanks John, I cannot use wild card here because I need to copy only those files that are there in the PARTITION list only. In that case, what change I am supposed to make in my above shell script? – arsenal Dec 24 '13 at 6:42
  • @TechGeeky Do you need to specify PARTITION as a bash array? Could you specify it just as easily as PARTITION=03579 ? – John1024 Dec 24 '13 at 6:48
  • Actually that partition has to be passed by Python script. So currently I am in the impression that I will be passing in the form of bash arrays and I haven't decided yet.. And in general number of elements in the bash arrays will be 420... And also if I need to do it one by one, then how I should do that as well? – arsenal Dec 24 '13 at 6:56
  • Python has much better looping and string handling constructs than bash. If this is all being controlled by a python script, I would have python create the scp command and feed it to subprocess for execution. – John1024 Dec 24 '13 at 7:04
  • Thanks a lot John. I will try to see whether I should do all these things in Python or not. And also, can you explain little bit on your solution for array? You are using IFS something which I am not aware of.. – arsenal Dec 24 '13 at 7:12

readonly PRIMARY=/data01/primary/.
readonly FILERS_LOCATION=(machineB machineC)
readonly MAPPED_LOCATION=/bat/data/snapshot

for i in 0 3 5 7 9
    scp david@${FILERS_LOCATION[0]}:$dir1/weekly_1980_${i}_200003_5.data PRIMARY || scp david@${FILERS_LOCATION[1]}:$dir2/weekly_1980_${i}_200003_5.data $PRIMARY

To efficiently copy many files at once, use rsync instead of scp. Rsync uses the SSH connection to transfer files remotely.

for el in "${PARTITION[@]}"; do
for machine in "${FILERS_LOCATION[@]}"; do
  rsync -a "${rsync_filter[@]}" --exclude='*' "david@$machine:$dir1/" "$PRIMARY/"

Rsync will skip copying if the files are already present with the same size and modification time (you can configure this).

Alternatively, forget all about the fact that they're remote copies. Mount the remote filesystem with SSHFS, and proceed as you would copy local files.

mkdir machineB machine C
sshfs david@machineB: machineB
sshfs david@machineC: machineC
for el in "${PARTITION[@]}"; do
  cp -p "machineB/$dir1/weekly_1980_${el}_200003_5.data" "$PRIMARY/" ||
  cp -p "machineC/$dir1/weekly_1980_${el}_200003_5.data" "$PRIMARY/"

You can even do one better and make a union mount of the two machines. A union mount joins two directory trees into one, such that accessing a file from the union tree tries inside one tree first, and if the file is not present tries the same location in the other tree instead.

mkdir union
mount -t aufs -o dirs=$PWD/machineB:$PWD/machineC machineB+machineC union
for el in "${PARTITION[@]}"; do
  cp -p "union/$dir1/weekly_1980_${el}_200003_5.data" "$PRIMARY/"

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.