7

I'm transferring about 9TB across my gigabit LAN. To do so as quickly as possible (i hope) I mounted the destination via NFS on the source and ran rsync across it.

Here is my mount options: x.x.x.x:/mnt on /mnt type nfs (rw,noatime,nodiratime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=x.x.x.x,mountvers=3,mountport=56548,mountproto=udp,local_lock=none,addr=x.x.x.x)

Here is my rsync command: rsync -avWH --progress ./ /mnt/

looking at nload, what i see, for a single file is speed that spikes up to 900MBps then down to numbers in the KBps range, then back up. Here is a graphic from nload where you can see that the transfer seems to stop, midfile. The files are all typically 5-6GB in size. MTU is 9000; switch is a cisco 3750x with plenty of backplane speed. These are esxi 6.7 guests on 2 different hosts. There are no other guests that contend for network resources.

Nload

This image is ONE file being sent

Basically, I'm hoping there is a setting I have wrong or something I can change to keep the transfer speed somewhat consistent.

CPU utilization on the source is approximately 10%, on the dest is approximately 10%. The strange thing is that on the destination, iotop shows 99% i/o (sometimes) from nfsd, the source shows 60-80% IO from rsync. These are 7200RPM WD red drives. w

  • Are source and destination systems Unix/Linux, and if so do you have a login on them both? – roaima Jun 19 '18 at 6:56
  • If this is a Linux system, it could be a problem from the filesystem cache implementation that is known to have idle times. – schily Jun 19 '18 at 7:26
  • @roaima yes, they're both linux and I have root on both. – driz Jun 19 '18 at 13:43
  • @schily the strange thing is, interesting, I hadn't noticed with wget but I will test again. – driz Jun 19 '18 at 13:44
9

Unfortunately just about the worst thing you can do is to use rsync across NFS. (Or to any remote filesystem that's mounted into the local system.) This switches off almost all of the efficiency enhancements for which rsync is known.

For this much data one of the fastest ways to transfer it between systems may be to dump it across an unencrypted connection without any consideration for what was already on the target system.

Once you have at least a partial copy the best option is to use rsync between the two hosts. This allows rsync to run one process on each host to consider and compare differences. (The rsync will completely skip files that have the same size and modification date. For other files the client and server components will perform a rolling checksum to determine which block(s) need still to be transferred.)

  1. Fast dump. This example uses no authentication or encryption at all. It does apply compression, though, which you can remove by omitting both -z flags:

    Run this on the destination machine to start a listening server:

    cd /path/to/destination && nc -l 50505 | pax -zrv -pe
    

    Run this on the source machine to start the sending client:

    cd /path/to/source && pax -wz . | nc destination_server 50505
    

    Some versions of nc -l may require the port to be specified with a flag, i.e. nc -l -p 50505. The OpenBSD version on Debian (nc.openbsd, linked via /etc/alternatives to /bin/nc) does not.

  2. Slower transfer. This example uses rsync over ssh, which provides authentication and encryption. Don't miss off the trailing slash (/) on the source path. Omit the -z flag if you don't want compression:

    rsync -avzP /path/to/source/ destination_server:/path/to/destination
    

You may need to set up SSH certificates to allow login to destination_server as root. Add the -H flag if you need to handle hard links.

  • 1
    I attempted your commands, with Z i'm throttled, so once this test transfer completed, i'll remove compression to fully utilize my link (GREAT Tool, how did i not know of it!) Anyway, the destination side needed to be nc -l -p 50505, but this appears to be working. it definitely doesn't seem any faster than my rsync over nfs, but the transfer is MUCH more stable. -- Removing the z options gave me a stable 800-900Mbps with no more transfer gaps!! – driz Jun 20 '18 at 14:19
  • One thing I noticed is that I love my permissions and ownership settings and presumably this won't follow hard-links. luckily, a post pax rsync should resolve the hard links but i think im on my own for permissions. ustar apparently also has a limit "pax: File is too long for ustar " it looks like pax -pe might take care of the ownership/mode issue – driz Jun 20 '18 at 14:30
  • @driz apologies, you're right about -pe. I'll amend the answer. Re nc -l -p I think there are a number of different (and slightly incompatible) versions of nc floating around. I'll flag it up in my answer but my version definitely works here. – roaima Jun 20 '18 at 15:06
  • no apologies necessary, this is so much faster than what I was doing. I'll have to go back to transfer the files that were too long/large for pax, but this nc blackmagic will likely help me out there too. – driz Jun 20 '18 at 15:13
  • 1
    @driz remember that rsync will "fill in the gaps", so if you've 9TB to copy but you've done, say, 8TB of it, the rsync command in my answer will just transfer the missing 1TB. It's inherently restartable and will only copy data that's missing from the destination. It will also fix up permissions, ownerships, etc., and even make the evening meal. (Well, maybe not the meal.) – roaima Jun 20 '18 at 15:18
5

It is far better to use rsync directly between two hosts if possible. Remember, rsync is built to optimise network IO at the cost of increased disk IO; when using rsync on an NFS filesystem, disk IO translates to network IO, so that is a very suboptimal solution. Also if rsync thinks that both source and destination is local, it will switch off the optimizations and transfer complete files every time, instead of using the differential algorithm that only sends the differences.

Say you have a 5GB file that only differs in 1% of the data between source and destination.

  • When transferring between hosts, rsync will checksum the source and destination files, and only transfer the difference; on the destination the file is recreated using the old file and the new data from the source, and then the old file is replaced.
  • When transferring locally, it makes no sense to checksum each file, meaning you'd have to read 2 x 5GB and write 1 x 5GB for the example file. By switching to whole file mode, rsync only needs to read 1 x 5GB and write 1 x 5GB. On local disks this makes complete sense, when one is NFS the network bandwidth shoots through the roof.

If you can use rsync directly to the host serving the NFS filesystem, then do that, you will see a big improvement in the performance.

  • Thanks for your answer! While this helped me from an understanding the issue perspective, it didn't help me with how to move forward, which is why I selected the other response. Thank you though! – driz Jun 20 '18 at 14:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.