I'm transferring about 9TB across my gigabit LAN. To do so as quickly as possible (i hope) I mounted the destination via NFS on the source and ran rsync across it.
Here is my mount options:
x.x.x.x:/mnt on /mnt type nfs (rw,noatime,nodiratime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=x.x.x.x,mountvers=3,mountport=56548,mountproto=udp,local_lock=none,addr=x.x.x.x)
Here is my rsync command:
rsync -avWH --progress ./ /mnt/
looking at nload, what i see, for a single file is speed that spikes up to 900MBps then down to numbers in the KBps range, then back up. Here is a graphic from nload where you can see that the transfer seems to stop, midfile. The files are all typically 5-6GB in size. MTU is 9000; switch is a cisco 3750x with plenty of backplane speed. These are esxi 6.7 guests on 2 different hosts. There are no other guests that contend for network resources.
This image is ONE file being sent
Basically, I'm hoping there is a setting I have wrong or something I can change to keep the transfer speed somewhat consistent.
CPU utilization on the source is approximately 10%, on the dest is approximately 10%. The strange thing is that on the destination, iotop shows 99% i/o (sometimes) from nfsd, the source shows 60-80% IO from rsync. These are 7200RPM WD red drives. w