Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have a below shell script from which I am trying to copy 5 files in parallel. I am running my below shell script on machineA which tries to copy the file from machineB and machineC.

If the file is not there in machineB, then it should be there in machineC for sure.

I am using GNU Parallel here to download five files in parallel.

#!/bin/bash

readonly PRIMARY=/tech01/primary
readonly FILERS_LOCATION=(machineB machineC)
readonly MEMORY_MAPPED_LOCATION=/techbat/data/be_t1_snapshot
PRIMARY_PARTITION=(550 274 2 546 278 6 558 282 10 554 286 14) # this will have more file numbers

dir1=/techbat/data/be_t1_snapshot/20140501

find "$PRIMARY" -mindepth 1 -delete

do_copy() {
  el=$1
  scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 david@${FILERS_LOCATION[0]}:$dir1/t1_weekly_1680_"$el"_200003_5.data $PRIMARY/. || scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 david@${FILERS_LOCATION[1]}:$dir1/t1_weekly_1680_"$el"_200003_5.data $PRIMARY/.
}
export -f do_copy
parallel -j 5 do_copy ::: "${PRIMARY_PARTITION[@]}"

Problem Statement:-

The problem I am facing with the above script is - It is not able to recognize ${FILERS_LOCATION[0]}, ${FILERS_LOCATION[1]}, $dir1 and $PRIMARY inside do_copy method? And I am not sure why?

If I try to print out like this inside do_copy method nothing is printed out?

  echo ${FILERS_LOCATION[0]}    
  echo ${FILERS_LOCATION[1]}

But if I print out same thing just above do_copy method, then it works fine?

Anything I am missing here?

Update:-

Below is the code I am using -

#!/bin/bash

export PRIMARY=/tech01/primary
export FILERS_LOCATION=(machineB machineC)
export MEMORY_MAPPED_LOCATION=/techbat/data/be_t1_snapshot
PRIMARY_PARTITION=(0 548 272 4 544 276 8 556 280)

export dir1=/techbat/data/be_t1_snapshot/20140501

find "$PRIMARY" -mindepth 1 -delete

do_copy() {
  el=$1
  scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 david@${FILERS_LOCATION[0]}:$dir1/t1_weekly_1680_"$el"_200003_5.data $PRIMARY/. || scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 david@${FILERS_LOCATION[1]}:$dir1/t1_weekly_1680_"$el"_200003_5.data $PRIMARY/.
}
export -f do_copy
parallel -j 8 do_copy ::: "${PRIMARY_PARTITION[@]}"

Another Update:-

This is what I got after running the below script -

#!/bin/bash

export PRIMARY=/tech01/primary
export FILERS_LOCATION=(slc4b03c-407d.stratus.slc.ebay.com chd1b02c-0db8.stratus.phx.ebay.com)
export MEMORY_MAPPED_LOCATION=/techbat/data/be_t1_snapshot
PRIMARY_PARTITION=(0 548 272 4 544)

export dir1=/techbat/data/be_t1_snapshot/20140501

find "$PRIMARY" -mindepth 1 -delete

 echo ${FILERS_LOCATION[0]}    
 echo ${FILERS_LOCATION[1]}

do_copy() {
  el=$1
  echo "scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 bullseye@${FILERS_LOCATION[0]}:$dir1/t1_weekly_1680_"$el"_200003_5.data $PRIMARY/. || scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 bullseye@${FILERS_LOCATION[1]}:$dir1/t1_weekly_1680_"$el"_200003_5.data $PRIMARY/."
}
export -f do_copy
parallel -j 3 do_copy ::: "${PRIMARY_PARTITION[@]}"

Output I got -

david@tvxdbx1143:/home/david$ ./scp_files5.sh
machineB
machineC
When using programs that use GNU Parallel to process data for publication please cite:

  O. Tange (2011): GNU Parallel - The Command-Line Power Tool,
  ;login: The USENIX Magazine, February 2011:42-47.

This helps funding further development; and it won't cost you a cent.

To silence this citation notice run 'parallel --bibtex' once or use '--no-notice'.

scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 david@:/techbat/data/be_t1_snapshot/20140501/t1_weekly_1680_0_200003_5.data /tech01/primary/. || scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 david@:/techbat/data/be_t1_snapshot/20140501/t1_weekly_1680_0_200003_5.data /tech01/primary/.
scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 david@:/techbat/data/be_t1_snapshot/20140501/t1_weekly_1680_548_200003_5.data /tech01/primary/. || scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 david@:/techbat/data/be_t1_snapshot/20140501/t1_weekly_1680_548_200003_5.data /tech01/primary/.
scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 david@:/techbat/data/be_t1_snapshot/20140501/t1_weekly_1680_272_200003_5.data /tech01/primary/. || scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 david@:/techbat/data/be_t1_snapshot/20140501/t1_weekly_1680_272_200003_5.data /tech01/primary/.
scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 david@:/techbat/data/be_t1_snapshot/20140501/t1_weekly_1680_4_200003_5.data /tech01/primary/. || scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 david@:/techbat/data/be_t1_snapshot/20140501/t1_weekly_1680_4_200003_5.data /tech01/primary/.
scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 david@:/techbat/data/be_t1_snapshot/20140501/t1_weekly_1680_544_200003_5.data /tech01/primary/. || scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 david@:/techbat/data/be_t1_snapshot/20140501/t1_weekly_1680_544_200003_5.data /tech01/primary/.
share|improve this question
    
Your edited version works for me. Please do: echo $SHELL –  Ole Tange May 5 at 18:34
    
/bin/bash this is what I got. –  Webby May 5 at 18:35
    
Prepend scp with 'echo' and see if you get the correct output. –  Ole Tange May 5 at 19:07
    
Do the examples in the tutorial work? –  Ole Tange May 5 at 19:13
    
@OleTange: I have updated the question with the scp output. It is not even making correct scp output. –  Webby May 5 at 19:33

2 Answers 2

up vote 2 down vote accepted

Try exporting them and removing the array as bash can not export arrays:

export PRIMARY=/data01/primary
export FILERS_LOCATION_1=machineB
export FILERS_LOCATION_2=machineC
export MEMORY_MAPPED_LOCATION=/bexbat/data/be_t1_snapshot

export dir1=/bexbat/data/be_t1_snapshot/20140501

Or simply put all the constant variables into the function:

#!/bin/bash

PRIMARY_PARTITION=(0 548 272 4 544 276 8 556 280)

PRIMARY=/data01/primary
find "$PRIMARY" -mindepth 1 -delete

do_copy() {
  el=$1

  PRIMARY=/data01/primary
  FILERS_LOCATION=(machineB machineC)
  MEMORY_MAPPED_LOCATION=/bexbat/data/be_t1_snapshot

  dir1=/bexbat/data/be_t1_snapshot/20140501

  scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 david@${FILERS_LOCATION[0]}:$dir1/t1_weekly_1680_"$el"_200003_5.data $PRIMARY/. || scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 david@${FILERS_LOCATION[1]}:$dir1/t1_weekly_1680_"$el"_200003_5.data $PRIMARY/.
}
export -f do_copy
parallel -j 8 do_copy ::: "${PRIMARY_PARTITION[@]}"

Depending on what kinds of files you are copying you should look into rsync -z instead of scp. And consider running parallel --bibtex once (as suggested by parallel).

share|improve this answer
    
I tried this and it doesn't work somehow? I have updated the question with the shell script details. May be I am doing something wrong? –  Webby May 5 at 18:20
2  
Apparently bash arrays can not be exported. Use FILERS_LOCATION_1 and FILERS_LOCATION_2 instead. –  mtm May 5 at 22:28
    
Actually Ole said that code worked for him. What versions of bash do you both have? bash --version –  mtm May 5 at 22:31
    
@mtm bash version I have is GNU bash, version 4.2.24(1)-release (x86_64-pc-linux-gnu) –  Webby May 5 at 23:37
    
It seems you need bash 4.3 to export arrays. stackoverflow.com/questions/5564418/… Follow mtm's suggestion. –  Ole Tange May 6 at 6:25

You have exported a function but not the variables you are trying to use directly in the function.

parallel will launch a new shell for each run of do_copy and in that shell the variables are interpreted and do not exist.

If the -s SERVER option is used the --env VAR option will copy VAR from the initial parallel shell to the remote environment the command is run in:

parallel -j 5 -S localhost --env do_copy --env PRIMARY --env FILERS_LOCATION do_copy ::: "${PRIMARY_PARTITION[@]}"

You may be able to get away with the above localhost hack as I can't see a simple way to implement your multiple server logic into the parallel server -S options (unless you are guaranteed one server won't have the file?)

The nicer way would be to export the variables as Ole suggested, or passing all the require values as parameters to the do_copy function

share|improve this answer
    
Thanks for suggestion. I tried running like this and still it doesn't work? parallel -j 5 --env PRIMARY --env FILERS_LOCATION --env dir1 do_copy ::: "${PRIMARY_PARTITION[@]}" Anything wrong? –  Webby May 5 at 18:28
    
--env is only for remote runnning. –  Ole Tange May 5 at 18:29
    
@OleTange: I see, and I tried your suggestion as well. Am I doing something wrong as well? –  Webby May 5 at 18:33
    
aahh forgot about that, noted. cheers –  mtm May 5 at 21:18
    
@mtm: I also tried Ole suggestion but that is also not working for me. I have updated the question as well. Can you take a look? –  Webby May 5 at 21:59

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.