3

I have a script which copies .lzo files from a remote server. The script will attempt to copy from each of three servers, and will send an email if one of them is down.

My script is below and it works as expected:

export status_dir=$(mktemp -t -d transfer.XXXXXX)
cleanup() { rm -rf "$status_dir"; }
trap cleanup 0 # automatically clean up on exit

do_Copy() {
  el=$1
  PRIMSEC=$2
  scp david@"$local_dc":"$dir3"/new_clients_2014_"$el"_200003_5.data.lzo "$PRIMSEC"/. \
    || { touch "$status_dir/local_down" && scp david@"$remote_dc_1":"$dir3"/new_clients_2014_"$el"_200003_5.data.lzo "$PRIMSEC"/.; } \
    || { touch "$status_dir/primary_down" && scp david@"$remote_dc_2":"$dir3"/new_clients_2014_"$el"_200003_5.data.lzo "$PRIMSEC"/.; } \
    || { touch "$status_dir/secondary_down"; exit 1; }
}

parallel -j 12 do_Copy {} $PRIMARY ::: "${PARTITION_DATA_1[@]}" &
parallel -j 12 do_Copy {} $SECONDARY ::: "${PARTITION_DATA_2[@]}" &
wait

[[ -e "$status_dir/local_down" ]] && \
   mailx -r "david@host.com" -s "$local_dc machine down" "david@host.com" \
     <<<"Local machine $local_dc was down, so copied from Primary Remote machine $remote_dc_1"

[[ -e "$status_dir/primary_down" ]] && \
   mailx -r "david@host.com" -s "$remote_dc_1 machine down" "david@host.com" \
     <<<"Primary Remote machine $remote_dc_1 was down, so copied from Secondary Remote machine $remote_dc_2"

[[ -e "$status_dir/secondary_down" ]] && \
   mailx -r "david@host.com" -s "All three machine's are down" "david@host.com" \
     <<<"All three machines are down. Exiting out."

Once the script above is finished, I run a second script to decompress all .lzo files and then delete them:

#!/bin/bash
set -e
export PRIMARY=/test01/primary
export SECONDARY=/test02/secondary

parallel lzop -dU -- ::: {"$PRIMARY","$SECONDARY"}/*.lzo

I would like to combine the two scripts. Instead of having to run a second one, have the first one also decompress and delete the .lzo files. I am assuming only the code block below will be modified, but I am not sure what I should add because of the tests I use to check if the machines are down.

do_Copy() {
  el=$1
  PRIMSEC=$2
  scp david@"$local_dc":"$dir3"/new_clients_2014_"$el"_200003_5.data.lzo "$PRIMSEC"/. \
    || { touch "$status_dir/local_down" && scp david@"$remote_dc_1":"$dir3"/new_clients_2014_"$el"_200003_5.data.lzo "$PRIMSEC"/.; } \
    || { touch "$status_dir/primary_down" && scp david@"$remote_dc_2":"$dir3"/new_clients_2014_"$el"_200003_5.data.lzo "$PRIMSEC"/.; } \
    || { touch "$status_dir/secondary_down"; exit 1; }
}

Is combining the scripts even my best option? Or should I continue doing what I was doing earlier and have two scripts? If I do it the way I am doing it now, will I need to have more disk space to accommodate the .lzo and the un-compressed .lzo file at same time and then delete later on?

2

Sure you can combine it. Whether a server was up or not is completely irrelevant. Whichever server you download from, the .lzo file will always be in the same place. Also, it makes no difference in terms of disk space needed whether you have one script or two. Why should it? In both cases, you are decompressing the same file. Just add the lines from your second script to the copy function:

do_Copy() {
  el=$1
  PRIMSEC=$2
  scp david@"$local_dc":"$dir3"/new_clients_2014_"$el"_200003_5.data.lzo "$PRIMSEC"/. \
    || { touch "$status_dir/local_down" && scp david@"$remote_dc_1":"$dir3"/new_clients_2014_"$el"_200003_5.data.lzo "$PRIMSEC"/.; } \
    || { touch "$status_dir/primary_down" && scp david@"$remote_dc_2":"$dir3"/new_clients_2014_"$el"_200003_5.data.lzo "$PRIMSEC"/.; } \
    || { touch "$status_dir/secondary_down"; exit 1; }
}

PRIMARY=/test01/primary
SECONDARY=/test02/secondary

parallel lzop -dU -- ::: {"$PRIMARY","$SECONDARY"}/*.lzo
  • Thanks terdon, yeah it doesn't make any difference if I have one or two. In this way I need to have all 300 .lzo files copied and then I am doing uncompressed and delete. I was thinking copy one file and then uncompressed and delete and keep going like this. I am not sure which approach will be better. And yeah I can definitely combine my second script in first script for sure. By this line Just add the lines from your second script to the copy function: you mean to say, I should add after wait line in my script right? Not between do_Copy and first parallel line? – user1950349 Oct 12 '15 at 16:20
  • @user1950349 just add so that it is executed after the do_copy is finished. If you want each file to be decompressed as soon as it gets here, just add the lzop -du command to the do_copy function, after the scp. – terdon Oct 12 '15 at 16:25
  • Which approach you think I should go for, keeping in mind I am copying 300 files max to disk? And also you mean to say I should add between do_Copy function and first parallel line? Or After wait line? – user1950349 Oct 12 '15 at 16:29
  • @user1950349 no, it must be after the do_copy, it needs to be after the file has been copied. Personally, I would decompress each file as it comes, so I would add the decompression command inside the do_copy function. But that's more a question of personal preference than anything else. Either way should be OK. – terdon Oct 12 '15 at 16:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.