I have a
linux server : a1 it's local disk is : /scratch a directory in local disk is : /scratch/builda
/builda has over 100K files and is about 100GB
My compute cluster has about 100 servers - a1 through a100
I need to copy a:/scratch/builda to a2 - a100 under the same structure. Copy required for ALL files under builda and need to keep structure same, permissions same, checksum same etc.
I have tried rscyn --recursive a1:/scratch/builda a2:/scratch (thro a100) I have tried scp I have tried to automate ftp
I have looked at using parallel with rsycn
all of these are slow or break down after 10 or 12 run - and then slow down - and then I can't predict when they complete.
I read about grid-ftp - but seems complicated and i don't quite get globus.
I need to do this on both Windows (b1:x:\builda to b2-b100:x:\builda) and on linux (a1:/scratch/builda to a2-a100:/scratch/builda) fast and password less (or with a ssh key)