Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I am trying to copy files from machineB and machineC into machineA as I am running my below shell script on machineA.

If the files is not there in machineB then it should be there in machineC for sure so I will try copying the files from machineB first, if it is not there in machineB then I will try copying the same files from machineC.

I am copying the files in parallel using GNU Parallel library and it is working fine. Currently I am copying two files in parallel.

Currently, I am copying the PRIMARY_PARTITION files in PRIMARY folder using GNU parallel and once that is done, then I copy the SECONDARY_PARTITION files in SECONDARY folder using same GNU parallel so it is sequential as of now w.r.t PRIMARY and SECONDARY folder

Below is my shell script and everything works fine -

#!/bin/bash

export PRIMARY=/test01/primary
export SECONDARY=/test02/secondary
readonly FILERS_LOCATION=(machineB machineC)
export FILERS_LOCATION_1=${FILERS_LOCATION[0]}
export FILERS_LOCATION_2=${FILERS_LOCATION[1]}
PRIMARY_PARTITION=(550 274 2 546 278) # this will have more file numbers
SECONDARY_PARTITION=(1643 1103 1372 1096 1369) # this will have more file numbers

export dir3=/testing/snapshot/20140103

# delete primary files first and then copy
find "$PRIMARY" -mindepth 1 -delete

do_CopyInPrimary() {
  el=$1
  scp david@$FILERS_LOCATION_1:$dir3/new_weekly_2014_"$el"_200003_5.data $PRIMARY/. || scp david@$FILERS_LOCATION_2:$dir3/new_weekly_2014_"$el"_200003_5.data $PRIMARY/.
}
export -f do_CopyInPrimary
parallel -j 2 do_CopyInPrimary ::: "${PRIMARY_PARTITION[@]}"

# delete secondary files first and then copy
find "$SECONDARY" -mindepth 1 -delete

do_CopyInSecondary() {
  el=$1
  scp david@$FILERS_LOCATION_1:$dir3/new_weekly_2014_"$el"_200003_5.data $SECONDARY/. || scp david@$FILERS_LOCATION_2:$dir3/new_weekly_2014_"$el"_200003_5.data $SECONDARY/.
}
export -f do_CopyInSecondary
parallel -j 2 do_CopyInSecondary ::: "${SECONDARY_PARTITION[@]}"

Problem Statement:-

Is there any way I can launch two threads, one to copy files in PRIMARY folder using the same setup as I have above, meaning it will copy two files in parallel. And second thread to copy the files in SECONDARY folder using the same setup as I have above, it should also copy two files parallel simultaneously?

Meaning they should copy files in parallel both in PRIMARY and SECONDARY folder simultaneously not once PRIMARY folder is done, then copy files in SECONDARY folder.

Currently, once PRIMARY folder file is done, then only I try copying the files in SECONDARY folder.

In short, I just need to launch two threads one thread will run this -

# delete primary files first and then copy
find "$PRIMARY" -mindepth 1 -delete

do_CopyInPrimary() {
  el=$1
  scp david@$FILERS_LOCATION_1:$dir3/new_weekly_2014_"$el"_200003_5.data $PRIMARY/. || scp david@$FILERS_LOCATION_2:$dir3/new_weekly_2014_"$el"_200003_5.data $PRIMARY/.
}
export -f do_CopyInPrimary
parallel -j 2 do_CopyInPrimary ::: "${PRIMARY_PARTITION[@]}"

And second thread will run this -

# delete secondary files first and then copy
find "$SECONDARY" -mindepth 1 -delete

do_CopyInSecondary() {
  el=$1
  scp david@$FILERS_LOCATION_1:$dir3/new_weekly_2014_"$el"_200003_5.data $SECONDARY/. || scp david@$FILERS_LOCATION_2:$dir3/new_weekly_2014_"$el"_200003_5.data $SECONDARY/.
}
export -f do_CopyInSecondary
parallel -j 2 do_CopyInSecondary ::: "${SECONDARY_PARTITION[@]}"

And once all the files are copied successfully, it should echo the message, that all the files are copied. In java, I know how to launch two threads and each thread is performing certain task but not sure how in bash shell script this will work?

My main task is to copy two files in parallel using GNU parallel in PRIMARY folder and SECONDARY folder at a same time?

Is this possible to do in bash shell script?

share|improve this question

3 Answers 3

up vote 3 down vote accepted

The obvious is:

parallel -j 2 do_CopyInPrimary ::: "${PRIMARY_PARTITION[@]}" &
parallel -j 2 do_CopyInSecondary ::: "${SECONDARY_PARTITION[@]}" &
wait

But this way the secondary does not wait for the primary to finish and it does not check if the primary was successful. Let us assume that $PRIMARY_PARTITION[1] corresponds to $SECONDARY_PARTITION[1] (so if you cannot read the file from $PRIMARY_PARTITION[1] you will read it from $SECONDARY_PARTITION[1] - That also means that $PRIMARY_PARTITION and $SECONDARY_PARTITION have the same number of elements). Then you can condition the running of $SECONDARY_PARTITION[1] on $PRIMARY_PARTITION[1].

do_Copy() {
  PRIMARY_PARTITION=(550 274 2 546 278) # this will have more file numbers
  SECONDARY_PARTITION=(1643 1103 1372 1096 1369) # this will have more file numbers
  pel=${PRIMARY_PARTITION[$1]}
  sel=${SECONDARY_PARTITION[$1]}
  do_CopyInPrimary $pel || 
    do_CopyInSecondary $sel || 
    echo Could not copy neither $pel nor $sel
}
export -f do_Copy
# Number of elements in PRIMARY_PARTITION == SECONDARY_PARTITION
seq ${#PRIMARY_PARTITION[@]} | parallel -j 2 do_Copy

This will get the dependency right, but it will only copy 2 at a time in total. With -j4 you risk running 4 primaries at the same time, so we need to guard against that, too:

do_Copy() {
  PRIMARY_PARTITION=(550 274 2 546 278) # this will have more file numbers
  SECONDARY_PARTITION=(1643 1103 1372 1096 1369) # this will have more file numbers
  pel=${PRIMARY_PARTITION[$1]}
  sel=${SECONDARY_PARTITION[$1]}
  sem -j2 --fg --id primary do_CopyInPrimary $pel || 
    sem -j2 --fg --id secondary do_CopyInSecondary $sel || 
    echo Could not copy neither $pel nor $sel
}
export -f do_Copy
# Number of elements in PRIMARY_PARTITION == SECONDARY_PARTITION
seq ${#PRIMARY_PARTITION[@]} | parallel -j 4 do_Copy

sem will limit the number of primaries to 2 and the number of secondaries to 2.

share|improve this answer
    
Thanks for the help. In your suggestion, you said Let us assume that $PRIMARY_PARTITION[1] corresponds to $SECONDARY_PARTITION[1]. What does this mean? And also in your last code example, you are passing PRIMARY_PARTITION array only? It does not take SECONDARY_PARTITION array I guess? Or may be I misunderstood some part of your answer? –  Webby May 19 at 15:03
    
Thanks for edit. I should have been more clear, sorry about that. In my case, number of elements in PRIMARY_PARTITION might be different as compared to SECONDARY_PARTITION. It will not be same at all. In this case, then how would I do that? –  Webby May 20 at 16:55

Bash doesn't support threading, but it does support background multi-processing. That is, the process is cloned into a new process space, with its own environment, working directory, etc, and all communication has to happen through normal IPC channels. But otherwise it looks a lot like threading.

You do this by "backgrounding" a code block. Like this:

#!/bin/bash
{
    echo "Foo"
    sleep 1
    echo "Foo: done"
}&    
echo "Bar"
sleep 1
echo "Bar: done"

output

Bar
Foo
**[1 second delay]**
Bar: done
Foo: done

You can get the same effect by wrapping a code block in a function, and running that function as a background job.

Alternately, you can wrap your code block in parentheses instead of braces. Statements in parentheses explicitly (and always) run in a separate process; normally statements in braces are grouped but run without forking. Running the code in the background by using the & suffix forces that code to run in a separate process.

share|improve this answer
    
Thanks for suggestion. In my example how will this work out? Can you provide an example for that as well if possible? –  Webby May 19 at 15:19
    
Can you provide an example if possible basis on my above solution? I am still learning shell script as of now. –  Webby May 21 at 4:52
    
@webby no. I try to only post content that is useful to the community at large rather than just a single user. If they're is a concept that you don't understand, then post a question about that concept. –  tylerl May 21 at 4:58

It would be a good idea to use rsync instead of scp for this. By copying all the files in one command instead of running scp for each file, it will save you a lot of time and effort, and will help ensure the data is copied correctly. It also will skip copying existing files from machineC. Something like this:

#!/bin/bash

files="one two three"
machines="machineB machineC"

for machine in machines
do
    ssh $machine -c "cd source_directory || exit 1; rsync -avPz --ignore-existing $files machineA:/receive_directory/"
done
share|improve this answer
    
You've changed the logic: each file should be copied from machineB if present and from machineC otherwise, but your script instead copies or not based on dates (and if you change the options to rsync, you'll still not be able to make the copy from machineC depend on the presence of the file on machineB). –  Gilles May 19 at 18:14
    
If the files on machineC are newer than the files on machineB, you'd be correct, but if the timestamps are the same, or if the files on machineB are newer, you'd be incorrect. Either way, the --ignore-existing rsync option would prevent a file from being recopied from machineC if it was already copied from machineB. Its man page says, "--ignore-existing skip updating files that exist on receiver," so it would make the copy from machineC depend entirely on the presence of the file on machineB. I've added --ignore-existing to the answer; thanks. –  blujay Jun 20 at 7:03

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.