NFSv3 vs NFSv4 performance drop

Question

I have two diskless systems setups. Both of them mount /home directory from the same server. The /home export is Kerberized and works without problems, only performance seems to be the issue. Everything is connected to the same switch, using the same cables, port configuration is the same, etc. fstab mount options are a bit different, because NFSv3 and NFSv4 have different subsets of them.

I copied about 25000 small files (smaller than 10kb) to /dev/shm/dir (to copy physically between the server and a client) on each client, and then rsynced them to my ~. Times differ by about 15 seconds. Does anyone know why this happens? Is NFSv4 slower than NFSv3? Or is there a specific option, that could improve the situation? I would greatly appreciate some help.

Server

Debian "Bullseye", 5.10.0-23-amd64 #1 SMP Debian 5.10.179-1 (2023-05-12)
NFS Export: /home 1.2.3.4/24(rw,async,no_subtree_check,sec=krb5:krb5i:krb5p)

# cat /proc/fs/nfsd/versions
-2 +3 +4 +4.1 +4.2

Client1 using NFSv3

Debian "Bullseye", 5.10.0-23-amd64 #1 SMP Debian 5.10.179-1 (2023-05-12)

# cat /etc/fstab
5.6.7.8:/home  /home   nfs rw,nodev,nosuid,hard,nolock,proto=tcp,nfsvers=3,sec=krb5    0   0

# findmnt
└─/home   5.6.7.8:/home nfs   rw,nosuid,nodev,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,nolock,proto=tcp,timeo=600,retrans=2,sec=krb5,mountvers=3,mountport=40531,mountproto=tcp,local_lock=all

[client1] /dev/shm ➽ $ time rsync --links -r dir ~/dir1/
real    0m33,835s
user    0m0,582s
sys 0m6,062s

Client2 using NFSv4.2

Debian "Bookworm", 6.1.0-11-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.38-4 (2023-08-08)

# cat /etc/fstab
5.6.7.8:/home  /home   nfs rw,nodev,nosuid,ac,hard,proto=tcp,nfsvers=4.2,sec=krb5    0   0

# findmnt
└─/home   5.6.7.8:/home nfs4   rw,nosuid,nodev,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=krb5,local_lock=none

[client2] /dev/shm ➽ $ time rsync --links -r dir ~/dir2/
real    0m48,155s
user    0m0,671s
sys 0m8,472s

Edit: Sep 10th, 11:53

I read some statistics from the mountstats. Additionally i mounted the /home with krb5p and it turned out that encryption does not impose any visible impact on the throughput. Metodology:

• cat /proc/self/mountstats > /dev/shm/mountstats.start
• run rsync
• mountstats mountstats --since /dev/shm/mountstats.start > nfs.mountstats.rsync.txt
• mountstats iostat --since /dev/shm/mountstats.start > nfs.iostat.rsync.txt

NFSv3 client 

           ops/s       rpc bklog
        1874.078           0.000

read:              ops/s            kB/s           kB/op         retrans    avg RTT (ms)    avg exe (ms)
                   0.000           0.000           0.000        0 (0.0%)           0.000           0.000
write:             ops/s            kB/s           kB/op         retrans    avg RTT (ms)    avg exe (ms)
                 159.312         228.017           1.431        0 (0.0%)           0.720           0.843

NFSv4.2 client

           ops/s       rpc bklog
        1436.392           0.000

read:              ops/s            kB/s           kB/op         retrans    avg RTT (ms)    avg exe (ms)  avg queue (ms)          errors
                   0.025           0.062           2.449        0 (0.0%)           0.500           0.500           0.000        0 (0.0%)
write:             ops/s            kB/s           kB/op         retrans    avg RTT (ms)    avg exe (ms)  avg queue (ms)          errors
                 129.076         201.433           1.561        0 (0.0%)           0.322           0.407           0.061        0 (0.0%)

Edit: Sep 11th, 9:33

I made additional NFS server using Debian "Bookworm" 12 and created a simple, non-Kerberized NFS 4.2 share on an nvme Drive. There is no difference, the same rsync took about 50 seconds. So, this is not an operating system/kernel/hardware issue apparently, but it is NFS-related.

Then, I mount the same share using NFSv3 and the rsync took about 26 seconds. So, is really NFSv4 that slow when compared to NFSv3?

Edit: Sep11, 18:29 (in response to @ron answer)

are you the only other one on the system (both nfs-server and nfs-client)

Currently I have two setups, one shared with other machines, but almost no one was using the share at the time of tests and the I/O was at idle levels. The second setup is private.

is there any other network load;

No more than "idle" traffic at the time of tests. We have 10 Gbps core fiber optics uplinks, workstations have 1 Gbps uplinks. HP Enterprise switches. To minimize network problems (we also do not want to push this traffic through the router) the NFS server and all workstations are in the same subnet and connected to the same switch. And this is not a new setup, I'm only trying to switch from NFSv3 to NFSv4.

You set up the network and have full control of the switches/routers, > or was that all set up and managed by people you've never met

I control the hardware and did the setup.

and also possibly along with the kerberos security setup... adding security to improve performance, said no one ever.

I tried both Kerberized and non-Kerberized mount, both showed that NFSv3 was faster than NFSv4.

I stumbled upon this by accident. We have almost 200 workstations, which boot diskless. Everything works fine in general. This year, when setting up the new system (newest Debian 12) using NFSv4 I noticed, that copying an unpacked archive (with a lot of small files, icons, some symlinks) is somewhat long. So I decided to check if copying takes about (I do not care about small variations) the same time on the previous NFSv3 system. It turned out that it takes longer on NFSv4. I tried with and without Kerberos, always using TCP (proto=tcp).

I thought that there is a lot of variables in the equation, so please let us forget for a moment about Kerberos, etc. I setup this private system:

• no Kerberos
• server exports a single directory located on an nvme drive
• client mount the exported directory, no kerberos, only nfsvers=4.2,ac,rw
• Both client and server are Dell workstations with 1 Gbps Ethernet
• Both client and server are connected to the same switch and are in the same subnet (VLAN), so no router in the network traffic
• Both client and server run the same operating system and kernel: Debian 12 "Bookworm" 6.1.0-12-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.52-1 (2023-09-07)

Then I did (several times) this simple test:

• Get the archive with files to copy to /dev/shm. This one: https://github.com/vinceliuice/Tela-icon-theme/archive/refs/tags/2023-06-25.tar.gz
• unpack the archive
• rsync --links -r unpacked_archive /path/to/mounted/nfsshare/dest

It took about 50 seconds.

• Then i remounted this share using NFSv3
• and repeated the same test

It took about 30 seconds.

So, same machines, same network, no kerberos, same system, same kernels, only NFS version is different -- and there is a big difference. It is interesting and I would like to know what is happening and learn something. At this moment getting all this into consideration I have no other answer than just NFSv4 is simply slower. I found very old article about NFSv4 performance and expected that a lot has changed since, but maybe not in this case?

NFSv4 file creation is actually about half the speed of file creation > over NFSv3, https://www.linux.com/news/benchmarking-nfsv3-vs-nfsv4-file-operation-performance/

In my case I'm creating a lot of small files on the mounted NFS share.

Why trying this with many small files is important? Because students use Git, large IDE's create many small files, sometimes hundreds and more, students compile a lot of software, etc. And there is a lot of students -- so thousands and thousands of small files.

Sep 12, 10:55 (test results)

We never had any issue with tranfering large files, we do it on a daily basis and speeds are about full uplink speed. But regardless I did the copying test with the large file. /nvme is a NFS share.

/dev/shm ➽ $ dd status=progress if=/dev/zero of=test.file bs=1M count=6000
3488612352 bytes (3,5 GB, 3,2 GiB) copied, 1 s, 3,5 GB/s
6000+0 records in
6000+0 records out
6291456000 bytes (6,3 GB, 5,9 GiB) copied, 1,80348 s, 3,5 GB/s

[orange14] /dev/shm ➽ $ rsync --verbose --progress test.file /nvme/test.file

sent 6.292.992.082 bytes  received 35 bytes  105.764.573,39 bytes/sec
total size is 6.291.456.000  speedup is 1,00

I also did several scp copies and noted times.

[orange14] /dev/shm ➽ $ time scp test.file /nvme/test.file6

real    0m59,463s
real    0m59,700s
real    0m59,780s

It is about 105 MB/s.

As for the test with the machines with larger amounts of RAM, I do not have spare servers now that I can use to this purpose, and at this moment I do not have that much time to spare on such detailed tests of NFS in general. I repeated some test for small files and plotted them. Both server and client were Dell workstations, the same OS, the same kernel, no Kerberos, just simple NFSv3 or NFSv4.2 share. I was copying the same archive over and over again.

Kind regards

Answer 1

... 25000 small files

my first inclination would be that general system overhead and file system I/O, even though it is not disk I/O because you are running from tmpfs, is the main culprit causing variability, and also possibly along with the kerberos security setup... adding security to improve performance, said no one ever.

But, unless you run a real-time kernel to rule out variability, you will see some variability. And then to conclude it is because NFS v3 vs v4 or any other NFS parameter is likely to be mostly not true and you will be chasing your tail.

You did not mention many other important things -

• are you the only other one on the system (both nfs-server and nfs-client)
• is there any other network load;
• the network layout and hardware involved and is there a possibility that is having an affect? You set up the network and have full control of the switches/routers, or was that all set up and managed by people you've never met

I am not familiar with debian, but in RHEL there is tuned and the default profile is throughput-performance which * Broadly applicable tuning that provides excellent performance across a variety of common server workloads.* Other profiles that may be of benefit are latency-performance, network-latency, or network-throughput. That's the best I can tell you there, you would need to research more into it : https://www.redhat.com/sysadmin/linux-tuned-tuning-profiles

https://www.redhat.com/sysadmin/linux-tuned-tuning-profiles

you said 25,000 files less than 10kb each. That's about 250MB total of stuff, in my opinion too small use as a reference and unless you have both systems identical in hardware and identical in operating system setup and configuration with the only variable being NFS v3 versus v4.0/4.1/4.2 you will be chasing your tail with numbers.

For what it's worth, I tried to understand if there's a performance difference between NFS v3 versus v4.0/4.1/4.2 and udp versus tcp for the mount protocol. Using RHEL 8.8 on identical servers, having xeon 24-core cpu's and 768gb of ram, with a single file of test.tar that was ~25gb in size being copied via rsync -P to show speed, and using tmpfs to help rule out disk i/o, I saw no difference in performance, and this was over 100gbps InfiniBand using a Mellanox switch with just me on the servers and network in a lab environment. All different NFS parameters were able to achieve an observed max of around 490 MB/s and an average around 470 MB/s after the few minutes of time it took to copy - what was interesting is that after a reboot the first time the copy could be as slow as 340 MB/s, but doing subsequent copies I would eventually peak at 490 MB/s. That was as far as I was willing to go and spend time on this to understand if I should or should not use NFS v3 with UDP versus v4.2 and TCP in a cluster setup; I did find proto=rdma instead of TCP netted somewhere between 5% and 15% speed improvement on my one file copy; so proto=rdma is best. On a 1gbps network all that i've found is on the nfs-server side async versus sync to cause any significant speed improvement, on infiniband it had no affect. Could some combination of your NFS parameters be causing NFS v4.2 to be slower than v3, possibly I don't know. But to ask is NFS v4.2 slower than v3 based on what I've tried to understand I would say no and per documented improvements of nfs v.2 from v3 it is supposed to be all around better. And there is also pnfs supported in nfs v4.

I have also found significant speed differences between RHEL 7.9 and 8.8 in regards to infiniband and NFS; a scp in RHEL 8.8 will hit 1.0 GB/s versus < 600 MB/sec in RHEL 7.9, and similar differences in NFS where RHEL 8.8 is better, which is nfs-utils-1.3 versus nfs-utils-2.whatever in rhel8.8. I don't know debian, bookworm or bullseye, if you're not running identical operating systems and the latest NFS version... all that can matter in my opinion. i would like to see more NFS v4.2 performance numbers published, as what an admin should expect to see to know if things are configured properly or if there's a problem.

recommendation for NFS speed testing first step:

• make a test.tar file; for a 1gbps network I would make it anywhere between 2 and 10 gigabyte
  • dd if=dev/zero of=test.tar bs=10G count=1
  • du -sh test.tar
• use rsync -P <source> <destination> to copy `test.tar to or from your nfs_server and nfs_client
  • observe the max reported speed
  • observe the avg reported speed when finished
  • watch the second and subsequent invoking of this, as the first can be first time overhead not related to NFS; you have to do 3 or more subsequent tries to be sure of the times.
• do a mount -t tmpfs -o size=100G tmpfs /scratch on both the nfs_server and nfs_client and have your test.tar located there when copying to or from each system to help rule out disk i/o. Adjust size accordingly, I'm assuming use of a server with 128GB more of RAM. You want this tmpfs larger than your test.tar obviously.
• exportfs -s on the server side to note the export options such sync or async
• mount on the client side to note the mount options such as nfs vers and proto and mountproto of udp|tcp|rdma
• note a scp secure copy transfer over ssh between the two systems as a sanity check, where it should happen at around 112 MB/sec on a 1gbps network, if not and is significantly lower than ~105 MB/sec it is not being slowed down by something else, so don't expect NFS to run at 100% capable speed. A 10gb test.tar at 112 MB/sec should take 89 seconds. I see this also via samba copied to a windows 10 pc.
• all this is keeping it simple to compare apples to apples, from here step 2 would be arranging a scenario where you have many files of a larger size versus many files of a smaller size.