5

We are using rsync to synchronize data from two NFS servers. One NFS server is on the east coast, the other is on west coast. RTT is about 110ms.

On the east coast NFS server I mount the west coasts NFS server mount point.

<server>:/home/backups on /mnt/backups type nfs4 (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=krb5,clientaddr=x.x.x.x,local_lock=none,addr=y.y.y.y).

The data is ALREADY on both servers and just to do a validation of the data (e.g. sync folders and when nothing needs to be changes). The following is how long it takes to validate that east coast server is the same as west cost server of a 7GB folder.

The follow takes about 8 minutes to complete over 7GB of data.

rsync -r -vvvv --info=progress2 --size-only /<local_path>/ /<remote_path>/

The following (which avoids using NFS mount) takes about 15seconds to complete over 7GB of data (same as above).

rsync -r -vvvv --info=progress2 --size-only /<local_path>/ <user>@<west_cost_NFS>:/<remote_path>/

again the above is NOT moving any data as the folders are already synchronized, its just validating the data is the same (based on size of files).

I've tried using -o async on client and in /etc/exports async on the server but the client won't ever show async when I run "mount" on the client. I assume async is default. I've tried changing rsize, wsize as well to larger values, but performance doesn't get much better. Am I just SOL on getting any better performance out of NFS?

10
  • 1
    I can't speak directly to your question, but this would seem to be an excellent application for a ZFS filesystem. Using ZFS snapshots would give you atomic updates of the slave server with no unnecessary data transfer.
    – Jim L.
    Apr 19, 2021 at 18:10
  • 1
    @JimL. While that would be an option, both of these NFS server are pretty "solidified" in their current operation. It would be a major hassle to convert them. I appreciate the recommendation though. Apr 19, 2021 at 18:13
  • 1
    Since rsync appears to be working perfectly, why try to use nfs at all? While I never did nfs stuff at the level you are doing, I know I largely gave up on it when it came to data syncs and backups because it was missing key file attributes that would make rsyncs fail when done over nfs, so using rsync directly seems to be the solution you've already found works perfectly?
    – Lizardx
    Apr 19, 2021 at 20:04
  • 1
    @Lizardx well its kind of self interest mainly because this isn't the only operation we do. So I figured speeding up this usecase would speed up other usecases as well. Apr 19, 2021 at 20:07
  • My understanding of nfs, which is weak, is that it's not really very robust, it's been limping along since Sun went away, and what I saw using it myself is that key things were not being handled, in particular new ext4 file system attributes, which it didn't support at all last I checked. There's probably some way to speed it up if the factor is 100x via straight rsync, but your macro experience mirrors roughly my micro experience, nfs is slow. And may have too many roots back in the 80s to really be fixed, though you never know, dedicates groups of motivated people can do wonders.
    – Lizardx
    Apr 19, 2021 at 20:22

3 Answers 3

8

It seems to me you're trying to use rsync wrong. Rsync's protocol is designed for the exact senario of comparing / synchronising large file systems on two separate servers. It does at much as it can locally on both the local and remote machine before comparing in the middle.

Its protocol is designed such that an rsync agent on one machine talks to an rsync agent on another and the protocol is designed to massively reduce the number of round trips (and total data) required to complete the task.

That is rsync is designed to work:

            [fast]        [slow SSH]        [fast]
File system <----> rsync <----------> rsync <----> File system

Rsync is optimised for network performance between the two agents, but it has no way to control the protocol used to access the disk. So when you mount a remote NFS file system you change the profile of network access:

            [fast]        [fast]        [slow NFS]
File system <----> rsync <------> rsync <---------> File system

Rsync can't do anything about this because it has absolutely no control over the NFS protocol.


One concrete difference here is that over NFS, every file must be individually requested. To explore a file tree containing /foo/bar/baz you have to request / [wait] then request /foo [wait] then request /foo/bar [wait] then finally request /foo/bar/baz. At 110ms latency per request that's 330ms latency and you only got one file.

Rsync's protocol between agents doesn't have this limitation. The agent running on the remote machine eagerly compiles a list of every file and directory in the remote file system being synchronised and sends over everything. There's only one request for the entire file tree!

See how rsync works

8
  • 1
    I mean I get that NFS is the slow weak spot, I was mainly trying to figure out how to speed up NFS for this purpose. Or if I'm missing something in general. Apr 19, 2021 at 21:04
  • 1
    @KevinVasko No NFS isn't slow. Rsync is ridiculously fast. You're simply not going to get close to what rsync over ssh can do. Apr 19, 2021 at 21:17
  • 1
    @KevinVasko Not true. Running over SSH kicks off another rsync process on the remote machine via SSH, the two proceed to talk to each other over the terminal. You can see it by running ps -ef | grep rsync on the remote machine while the sync is active. Apr 19, 2021 at 21:28
  • 1
    The nfs remote mount isn't on the local machine, it's on the remote machine, that's what some of the responses and comments are saying if I'm not misreading it. This answer covers that issue. Or rather, the mount itself is on the local machine, but the files aren't, all you've done is a mount a remote file system on the local machine, after that, all the issues happen, since the file systems are on two different machines, and as the answers note, that's going to be really slow with nfs compared to rsync to rsync, which is totally different than rsync to remote mounted nfs file system..
    – Lizardx
    Apr 19, 2021 at 21:29
  • 1
    @PhilipCouling Interesting, didn't know it worked that way. Apr 20, 2021 at 13:14
1

Your premise is wrong. When you perform a filesystem compare over NFS you are moving a lot of data - metadata about the files. For a large tree that's a lot of individual requests, each of which has latency.

When you use rsync over an SSH connection you're sending a stream of filenames and metadata for the remote side to validate. It may be the same number of files, and therefore the same amount of metadata, but it's streamed and therefore has very low overall latency.

For a 110ms RTT you could easily end up with 15 seconds vs 8 minutes.

Oh, when you start to use rsync replace the -r flag with -a (ideal) or -rt (sufficient). Unless you include file timestamps rsync will end up considering files on each end of the connection to be out of date with respect to each other.

5
  • So there is there an option for handling this or is my solution over ssh the appropriate solution? I am using --size-only flag because the destination machines have the sticky bit set to shift owner of the group/files to a different user. Using -a wouldn't work for us, `-t I don't think it copied over the correct timestamp when I initially did it so those are probably are different based on when it was "written". Apr 19, 2021 at 20:46
  • You're already using rsync so continue to do so. Just make sure you're using the client/server model (i.e. the one with ssh). Use rsync -rtv /<local_path>/ /<remote_path>/ so you maintain file times. Apr 22, 2021 at 22:44
  • Don't use --size-only as you'll get burned by echo hello >file (not) being updated with echo world >file. Apr 22, 2021 at 22:58
  • Will -rtv update the time without retransferring the entire file? It’s 120TB of data and not really wanting to wait on a resync. Apr 22, 2021 at 23:01
  • If you can guarantee no mismatches like the one I've shown then you should find that a one-off run that includes --size-only will update the metadata without validating the content. Try first with --dry-run and let it run for a few files Apr 22, 2021 at 23:27
0

using RHEL 8.7 currently (Feb 2023) having been using RHEL 7.9 previously, I can tell you that in RHEL 7.9 that I could never get NFS vers=4.2 to work. It would max out at vers=4.1. and only if I did not modify the /etc/nfs/nfs.conf or the /etc/sysconfig/nfs files from their original installed state.

This article mentions Some reviews of NFSv4 and NFSv4.1 suggest that these versions have limited bandwidth and scalability and that NFS slows down during heavy network traffic. The bandwidth and scalability issue is reported to have improved with NFSv4.2. This was last updated in April 2022.

https://www.techtarget.com/searchenterprisedesktop/definition/Network-File-System

I also asked this recently fwiw

Is there any reason to use NFS 3 over version 4.2?

So in my opinion from what I've experienced playing with NFS lately that unless you can get vers=4.2 and proto=rdma to work that maybe using nfs v3 as UDP might give the best performance. And especially specifying async in /etc/exportfs on the nfs server. You are correct you will not see async mentioned on the nfs client side as a mount option; I've tested this and observed a change from 50 MB/sec to 100 MB/sec immediately using rsync -P test_60gb.tar /nfsmount by modifying /etc/exports and then doing exportfs -arv. Having proto=tcp with sync is definitely bad for performance with NFS.

I have not had a chance to try proto=udp yet, in fact have had a hard time trying to make it happen.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .