I have a

linux server : a1 it's local disk is : /scratch a directory in local disk is : /scratch/builda

/builda has over 100K files and is about 100GB

My compute cluster has about 100 servers - a1 through a100

I need to copy a:/scratch/builda to a2 - a100 under the same structure. Copy required for ALL files under builda and need to keep structure same, permissions same, checksum same etc.

I have tried rscyn --recursive a1:/scratch/builda a2:/scratch (thro a100) I have tried scp I have tried to automate ftp

I have looked at using parallel with rsycn

all of these are slow or break down after 10 or 12 run - and then slow down - and then I can't predict when they complete.

I read about grid-ftp - but seems complicated and i don't quite get globus.

Any help.

I need to do this on both Windows (b1:x:\builda to b2-b100:x:\builda) and on linux (a1:/scratch/builda to a2-a100:/scratch/builda) fast and password less (or with a ssh key)


  • it needs to be automated - may be like once a day at 2PM when the build completes. – user3723896 Sep 12 '17 at 1:53
  • write a shell script, To automate everything – SuperKrish Sep 15 '17 at 4:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.