Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I am using scp to copy the files in parallel using GNU parallel with my below shell script and it is working fine.

I am not sure how can I use rsync in place of scp in my below shell script. I am trying to see whether rsync will have better performance as compared to scp or not in terms of transfer speed.

Below is my problem description -

I am copying the files from machineB and machineC into machineA as I am running my below shell script on machineA.

If the files is not there in machineB then it should be there in machineC for sure so I will try copying the files from machineB first, if it is not there in machineB then I will try copying the same files from machineC.

I am copying the files in parallel using GNU Parallel library and it is working fine. Currently I am copying five files in parallel both for PRIMARY and SECONDARY.

Below is my shell script which I have -

#!/bin/bash

export PRIMARY=/test01/primary
export SECONDARY=/test02/secondary
readonly FILERS_LOCATION=(machineB machineC)
export FILERS_LOCATION_1=${FILERS_LOCATION[0]}
export FILERS_LOCATION_2=${FILERS_LOCATION[1]}
PRIMARY_PARTITION=(550 274 2 546 278) # this will have more file numbers
SECONDARY_PARTITION=(1643 1103 1372 1096 1369 1568) # this will have more file numbers

export dir3=/testing/snapshot/20140103

do_Copy() {
  el=$1
  PRIMSEC=$2
  scp david@$FILERS_LOCATION_1:$dir3/new_weekly_2014_"$el"_200003_5.data $PRIMSEC/. || scp david@$FILERS_LOCATION_2:$dir3/new_weekly_2014_"$el"_200003_5.data $PRIMSEC/.
}
export -f do_Copy

parallel --retries 10 -j 5 do_Copy {} $PRIMARY ::: "${PRIMARY_PARTITION[@]}" &
parallel --retries 10 -j 5 do_Copy {} $SECONDARY ::: "${SECONDARY_PARTITION[@]}" &
wait

echo "All files copied."

Is there any way of replacing my above scp command with rsync but I still want to copy 5 files in parallel both for PRIMARY and SECONDARY simultaneously?

This is what I have tried so far. After reading some of the tutorials for rsync -

for n in "${PRIMARY_PARTITION[@]}"
do
    primary_files="$primary_files :$dir3"/new_weekly_2014_"$n"_200003_5.data
done

for n in "${SECONDARY_PARTITION[@]}"
do
    secondary_files="$secondary_files :$dir3"/new_weekly_2014_"$n"_200003_5.data
done

rsync -avz david@$FILERS_LOCATION_1"${primary_files}" $PRIMARY/ 2>/dev/null
rsync -avz david@$FILERS_LOCATION_2"${primary_files}" $PRIMARY/ 2>/dev/null

rsync -avz david@$FILERS_LOCATION_1"${secondary_files}" $SECONDARY/ 2>/dev/null
rsync -avz david@$FILERS_LOCATION_2"${secondary_files}" $SECONDARY/ 2>/dev/null 

But not sure how do I fit this in my above GNU Parallel shell script so that I can copy five files in parallel both for PRIMARY and SECONDARY simultaneously?

share|improve this question
    
You are searching an optimization. I miss what do you want to optimize. Do you have a bottleneck on network bandwidth, on the cpu time of server(A) or of clients (B or C)? On the total process time? Note that if the same file exists on the server and on the clients or on both clients, rsync will perform actions to decide the most recent one (following the rules you will give it). It means you will use resources. –  Hastur May 24 at 18:54

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.