1

I'm trying to rsync approx 10 TB of data from a remote system to the local machine, and I want to use the parallel utility for multi-thread execution. I want to trigger the rsync from the local server. Can someone please suggest how I may do this?

  • The bottleneck with rsync is often IO requests. Running multiple rsyncs in parallel will only make this worse. – wurtel May 24 at 8:49
  • parallel and xargs might help – ranisalt Jun 2 at 9:24
1

Can you elaborate on why this does not work:

seq -w 0 99 | parallel rsync -Havessh fooserver:src/*{}.png destdir/

From https://www.gnu.org/software/parallel/man.html#EXAMPLE:-Parallelizing-rsync

0

This has worked well for me:

cat /tmp/dir_list.txt | parallel --will-cite -j6 time rsync -ahrzp user@:/{} /

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.