3

I am trying to parallelize my rsync calls using gnu-parallel as shown below. But whenever I run my below script, it looks like it doesn't copy anything at all. But I do see lot of rsync processes (ps aux | grep rsync) running so not sure what's wrong here:

export PRIMARY=/data01/test_primary
export SECONDARY=/data02/test_secondary
export dir3=/bat/data/snapshot/20180227
PRIMARY_FILES=(685 959 682 679 688 651 909 906 657 881 884 878 853 707 847)
SECONDARY_FILES=(950 883 887 890 1001 994 997 1058 981 833)

export LOCATION_1="machineA"
export LOCATION_2="machineB"
export LOCATION_3="machineC"

do_Copy() {
  el=$1
  PRIMSEC=$2
  rsync -az golden@"$LOCATION_1":"$dir3"/proc_"$el"_5.data "$PRIMSEC"/. || rsync -az golden@"$LOCATION_2":"$dir3"/proc_"$el"_5.data "$PRIMSEC"/. || rsync -az golden@"$LOCATION_3":"$dir3"/proc_"$el"_5.data "$PRIMSEC"/. || exit 1
}
export -f do_Copy
parallel -j 5 do_Copy {} $PRIMARY ::: ${PRIMARY_FILES[@]} &
parallel -j 5 do_Copy {} $SECONDARY ::: ${SECONDARY_FILES[@]} &
wait

echo "All copied."

Is there anything wrong I am doing here?

1

You are processing the arguments incorrectly in the do_Copy function that you have defined. Take extra pre-caution in processing positional arguments in shell scripting. Your code just leave one argument from the array in your final command.

You need to use $@ as the complete list of arguments in your function as

do_Copy() {
  (( "$#" )) || { printf 'insufficient args supplied' >&2 ; return 2; }

  el="$1"; shift
  PRIMSEC="$@"

  rsync -az golden@"$LOCATION_1":"$dir3"/proc_"$el"_5.data "$PRIMSEC"/. || rsync -az golden@"$LOCATION_2":"$dir3"/proc_"$el"_5.data "$PRIMSEC"/. || rsync -az golden@"$LOCATION_3":"$dir3"/proc_"$el"_5.data "$PRIMSEC"/. || exit 1
}

and quote the variables when calling the array

export -f do_Copy
parallel -j 5 do_Copy {} "$PRIMARY" ::: "${PRIMARY_FILES[@]}" &

Once you store the $1 in the function, with a first call to shift the rest of the positional arguments are available under "$@" which you originally intended to store.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.