How to copy files from local datacenter to local datacenter and have some failover cases?

Question

I have a script which knows about on which HOSTNAME it will be running as it has the HOSTNAME variable which tells about this and it is being passed from outside. I want to check which datacenter that HOSTNAME is in and basis on that, I will copy the files from the particular machine.

• If app server HOSTNAME is in PHX, then it should copy files from holdermachine.phx.host.com.
• If app server HOSTNAME is in SLC, then it should copy files from holdermachine.slc.host.com.
• If app server HOSTNAME is in LVS, then it should copy files from holdermachine.lvs.host.com.

Now for failover case if any of the holdermachine goes down then script should copy from other datacenter holdermachine (can be chosen randomly) but as soon as that local datacenter holdermachine is back up, it should start copying from their local one instead of remote datacenter holdermachine. Also if possible, send an email that this holdermachine is down and not responding so starting to copy from remote holdermachine.

As of now below script will only copy files from HOLDER_LOCATION_phx and I got confuse how to make the above logic work.

#!/bin/bash

export PRIMARY=/test01/primary # copy PRIMARY_PARTITION into this folder
export SECONDARY=/test02/secondary # copy SECONDARY_PARTITION into this folder

readonly HOLDER_LOCATION_phx=(holdermachine.phx.host.com) # we might have more machines in future
readonly HOLDER_LOCATION_slc=(holdermachine.slc.host.com) # we might have more machines in future
readonly HOLDER_LOCATION_lvs=(holdermachine.lvs.host.com) # we might have more machines in future

export HOLDER_LOCATION_1=${HOLDER_LOCATION_phx[0]}
export HOLDER_LOCATION_2=${HOLDER_LOCATION_slc[0]}
export HOLDER_LOCATION_3=${HOLDER_LOCATION_lvs[0]}

PRIMARY_PARTITION=(550 274 2 546 278) # this will have more file numbers and it is being passed from outisde
SECONDARY_PARTITION=(1643 1103 1372 1096 1369 1568) # this will have more file numbers and it is being passed from outisde

export FILE_LOCATION=/batch/data/pk_snapshot
readonly HOSTNAME=$hostname # this is the hostname on which this script will be running where we are copying the files.
readonly FILE_TIMESTAMP=$file_timestamp

export dir3=$FILE_LOCATION/$FILE_TIMESTAMP

# I need to delete before copying the files.
find "$PRIMARY" -mindepth 1 -delete
find "$SECONDARY" -mindepth 1 -delete

do_Copy() {
  el=$1
  PRIMSEC=$2
  scp david@$HOLDER_LOCATION_1:$dir3/new_weekly_2014_"$el"_200003_5.data $PRIMSEC/. || scp david@$HOLDER_LOCATION_2:$dir3/new_weekly_2014_"$el"_200003_5.data $PRIMSEC/. || scp david@$HOLDER_LOCATION_3:$dir3/new_weekly_2014_"$el"_200003_5.data $PRIMSEC/.
}
export -f do_Copy

# copying 10 files in parallel simultaneously in primary and secondary folder
parallel --retries 10 -j 10 do_Copy {} $PRIMARY ::: "${PRIMARY_PARTITION[@]}" &
parallel --retries 10 -j 10 do_Copy {} $SECONDARY ::: "${SECONDARY_PARTITION[@]}" &
wait    

echo "All files copied."

Now what I need to do is -

• Check whether the HOSTNAME is in PHX or SLC or LVS. And then basis on that start copying from local datacenter
• But if the local datacenter holder machine is down then copy from remote datacenter holder machine and send out an email about this. Also if local datacenter machine is backup, then start copying from local one instead of remote datacenter one.

Our app server machine name will be like this. The only thing we need to check is .phx. or .slc. or .lvs. portion. Also it is not necessary that phx, slc and lvs will always be at same location bcoz sometimes machine name has some extra stuff and then followed by .phx. or .slc. or .lvs., so we need to check it properly.

appservermachineA.phx.host.com
appservermachineB.slc.host.com
appservermachineC.lvs.host.com

Answer 1

Pulling out the foo.bar.baz.code.provider.com part can be done fairly easily in a couple of ways. For example:

datacenter="$(echo "$HOSTNAME" | rev | cut -d. -f3 | rev)"

If your data center names don't match a simple pattern like that, you could use a case statement, and match based on full shell patterns:

case "$HOSTNAME" in
    *phx.provider.com)       DATACENTER="dc1" ;;
    *lax.otherprovider.com)  DATACETNER="dc2" ;;
    *.weirdness.*)           DATACENTER="dc3" ;;
    # ⋮
esac

For your failover case, you need to decide how you're going to determine a machine is down. The simplest would be to try a different machine if scp returns a non-zero exit status. That's what you're doing with ||.

(You have some other things there too—for example, I'm pretty sure that parallel call won't work, because it works on commands, not shell functions)

Answer 2

You could do something like this, based on @derobert's answer:

#!/usr/bin/env bash

primary="/test01/primary" # copy primary_partition into this folder
secondary="/test02/secondary" # copy secondary_partition into this folder

holder_location_phx=("holdermachine.phx.host.com") # we might have more machines in future
holder_location_slc=("holdermachine.slc.host.com") # we might have more machines in future
holder_location_lvs=("holdermachine.lvs.host.com") # we might have more machines in future

primary_partition=(550 274 2 546 278) # this will have more file numbers and it is being passed from outisde
secondary_partition=(1643 1103 1372 1096 1369 1568) # this will have more file numbers and it is being passed from outisde

file_location="/batch/data/pk_snapshot"
file_timestamp="$file_timestamp"

dir3="$file_location"/"$file_timestamp"

# I need to delete before copying the files.
find "$primary" -mindepth 1 -delete
find "$secondary" -mindepth 1 -delete

## Find where we are and choose the primary and alternative
## targets accordingly.
case "$HOSTNAME" in
    *phx.host.com) 
        datacenter=("${holder_location_phx[@]}")
        alternative1=("${holder_location_slc[@]}")
        alternative2=("${holder_location_lvs[@]}")
        ;;
    *lax.host.com) 
        datacenter=("${holder_location_lax[@]}")
        alternative1=("${holder_location_phx[@]}")
        alternative2=("${holder_location_lvs[@]}")
        ;;

    *lvs.host.com) 
        datacenter=("${holder_location_lvs[@]}")
        alternative1=("${holder_location_slc[@]}")
        alternative2=("${holder_location_phx[@]}")
        ;;

    *) echo "uknown host, exiting." && exit 1 ;;
    # ⋮
esac

do_copy() {
  el=$1
  primsec=$2
  scp david@"${datacenter[0]}":"$dir3"/new_weekly_2014_"$el"_200003_5.data "$primsec"/ || scp david@"${alternative1[0]}":"$dir3"/new_weekly_2014_"$el"_200003_5.data "$primsec"/ || scp david@"${alternative2[0]}":"$dir3"/new_weekly_2014_"$el"_200003_5.data "$primsec"/
}

export -f do_copy

# copying 10 files in parallel simultaneously in primary and secondary folder
parallel --retries 10 -j 10 do_copy {} "$primary" ::: "${primary_partition[@]}" &
parallel --retries 10 -j 10 do_copy {} "$secondary" ::: "${secondary_partition[@]}" &
wait    

echo "all files copied."

Note that I am not at all sure that your parallel call will work. I have also fixed some other issues in your script and simplified it a bit. I have kept the array structure since you mentioned that you might want to expand this to more servers later. Note that if you do so, you will need to loop over the server names stored in the arrays. For the moment, I am simply using the first element in each of them (e.g. ${datacenter[0]}).