1

I am running my shell script on machineA which copies the files from machineB and machineC to machineA.

If the file is not there in machineB, then it should be there in machineC for sure. So I will try to copy file from machineB first, if it is not there in machineB then I will go to machineC to copy the same files.

In machineB and machineC there will be a folder like this YYYYMMDD inside this folder -

/data/pe_t1_snapshot

So whatever date is the latest date in this format YYYYMMDD inside the above folder - I will pick that folder as the full path from where I need to start copying the files -

so suppose if this is the latest date folder 20140317 inside /data/pe_t1_snapshot then this will be the full path for me -

/data/pe_t1_snapshot/20140317

from where I need to start copying the files in machineB and machineC. I need to copy around 400 files in machineA from machineB and machineC and each file size is 2.5 GB.

Earlier, I was trying to copy the files one by one in machineA which is really slow. Is there any way, I can copy "three" files at once in machineA using threads in bash shell script?

Below is my shell script which copies the file one by one in machineA from machineB and machineC.

#!/usr/bin/env bash

readonly PRIMARY=/export/home/david/dist/primary
readonly FILERS_LOCATION=(machineB machineC)
readonly MEMORY_MAPPED_LOCATION=/data/pe_t1_snapshot

PRIMARY_PARTITION=(0 548 272 4 544 276 8 556 280 12 552 284 16 256 564 20 260 560 24 264 572) # this will have more file numbers around 200

dir1=$(ssh -o "StrictHostKeyChecking no" david@${FILERS_LOCATION[0]} ls -dt1 "$MEMORY_MAPPED_LOCATION"/[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9] | head -n1)
dir2=$(ssh -o "StrictHostKeyChecking no" david@${FILERS_LOCATION[1]} ls -dt1 "$MEMORY_MAPPED_LOCATION"/[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9] | head -n1)

## Build your list of filenames before the loop. 
for n in "${PRIMARY_PARTITION[@]}"
do
    primary_files="$primary_files :$dir1"/t1_weekly_1680_"$n"_200003_5.data
done

if [ "$dir1" = "$dir2" ]
then
    find "$PRIMARY" -mindepth 1 -delete
    rsync -avz david@${FILERS_LOCATION[0]}"${primary_files}" $PRIMARY/ 2>/dev/null
    rsync -avz david@${FILERS_LOCATION[1]}"${primary_files}" $PRIMARY/ 2>/dev/null
fi  

So I am thinking instead of copying one file at a time, why not just copy "three" files at once and as soon these three files are done, I will move to another three files in the list to copy at same time?

I tried opening three putty instances and was copying one file from those three instances at the same time. All the three files were copied in ~50 seconds so that was fast for me. And because of this reason, I am trying to copy three files at once instead of one file at a time.

Is this possible to do? If yes, then can anyone provide an example on this? I just wanted to give a shot and see how this is working out.

@terdon helped me with the above solution but I wanted to try copying three files at once to see how it will behave.

Update:-

Below is the simplified version of the above shell script. It will try to copy files from machineB and machineC into machineA as I am running the below shell script on machineA. It will to try copy file numbers which are present in PRIMARY_PARTITION.

#!/usr/bin/env bash

readonly PRIMARY=/export/home/david/dist/primary
readonly FILERS_LOCATION=(machineB machineC)
readonly MEMORY_MAPPED_LOCATION=/data/pe_t1_snapshot

PRIMARY_PARTITION=(0 548 272 4 544 276 8 556 280 12 552 284 16 256 564 20 260 560 24 264 572) # this will have more file numbers around 200

dir1=/data/pe_t1_snapshot/20140414
dir2=/data/pe_t1_snapshot/20140414

## Build your list of filenames before the loop. 
for n in "${PRIMARY_PARTITION[@]}"
do
    primary_files="$primary_files :$dir1"/t1_weekly_1680_"$n"_200003_5.data
done

if [ "$dir1" = "$dir2" ]
then
    # delete the files first and then copy it.
    find "$PRIMARY" -mindepth 1 -delete
    rsync -avz david@${FILERS_LOCATION[0]}"${primary_files}" $PRIMARY/
    rsync -avz david@${FILERS_LOCATION[1]}"${primary_files}" $PRIMARY/
fi
7
  • Don't parse ls output.
    – l0b0
    Apr 21, 2014 at 8:41
  • Can you simplify the code some more? There's a lot of stuff there that's not relevant to the problem: redirection, removing existing files, readonly declarations...
    – l0b0
    Apr 21, 2014 at 8:45
  • Why three files simultaneously? Don't you mean N hosts simultaneously? Copying N files simultaneously from one machine would be slower than sequentially.
    – l0b0
    Apr 21, 2014 at 8:47
  • @I0b0 no, he cant simplify it - he doesnt know what any of it does. That mish-mash was written by unconcerned third-parties on this site throughout a series of questions, and apparently handles the backup/restore operations for entire datacenters but he doesnt know what it does. Just hope those databases dont handle your credit card information.
    – mikeserv
    Apr 21, 2014 at 16:49
  • @mike. whatever you are thinking, its not right at all. These scripts are not doing whatever you are thinking, they are just just copying the files. And I already know what these are doing.. I was not able to reply since I was away since 7-8 hours. And also if you see my question, I already mentioned terdon helped me with this so its not I am hiding anything. You are just rude w.r.t my post always.. Shell scipting is not my first language but m learning.. l0b0 I will edit in couple of minutes..
    – arsenal
    Apr 21, 2014 at 16:56

2 Answers 2

1

Doing several copies in parallel is rarely useful: whether the limiting factor is network bandwidth or disk bandwidth, you'll end up with N parallel streams, each going at 1/N times the speed.

On the other hand, when you're copying from or to multiple sources (here B and C), then there is an advantage to doing the copies in parallel if the bottleneck is in on the side of B and C (rather than on the common side). So you can try doing the copies in parallel:

rsync -avz david@${FILERS_LOCATION[0]}"${primary_files}" $PRIMARY/ &
rsync -avz david@${FILERS_LOCATION[1]}"${primary_files}" $PRIMARY/ &
wait

Note that the output from the two rsync commands will be intermixed; you may want to send it to separate files.

log_base=$(date +%Y%m%d-%H%M%S-$$)
rsync -avz david@${FILERS_LOCATION[0]}"${primary_files}" $PRIMARY/ >$log_base-B.log &
rsync -avz david@${FILERS_LOCATION[1]}"${primary_files}" $PRIMARY/ >$log_base-C.log &
wait

You're using several SSH connections to the same destination in your script. Establishing an SSH connection has an unavoidable latency. You can save a bit of time by leaving the connection open and reusing it, which is easy thanks to master connections.

1
  • Thanks. Sorry I was away for a while. I just ran your suggestion but somehow it was copying pretty slowly. I know you suggested it won't be useful in your example. And also, if we copy the files in parallel as you have suggested above, then how many files it will try to copy in parallel? Can we copy three or two files in parallel and as soon as those are done, then move to another files? Is this possible to do? I am suspecting with the way you have suggested, it will try copy as many files as possible in one shot.
    – arsenal
    May 3, 2014 at 19:51
0

You can do multiple things in parallel by using background processes. As a general example:

rsync foo machine1: &
rsync bar machine2: &
rsync baz machine3: &
wait

The wait ensures that the program does not progress beyond that point before all the background processes created by the script are finished.

The & goes at the very end of each command line, since like ; it's a command separator.

1
  • Thanks. I already know that I can use & but how do I use & in my above example in such a way such that I can download three files at once? That's my question as it is causing me lot of confusion.. My background is not from shell scripting so having little bit of problem. If you can provide an example basis on my question then it will be of great help to me.
    – arsenal
    Apr 21, 2014 at 7:53

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .