9

I have a FreeBSD storage Server exporting ZFS Datasets via NFS, but the performance for small file transfers is below the acceptable limit.

Some background info:

  1. The exports are handeled via /etc/exports, not by the sharenfs property in ZFS.
  2. The Server has 128GB RAM and 2x8 core CPU's
  3. The NFS server is configured to use 192 threads (vfs.nfsd.maxthreads: 192)
  4. The zpool spans 4x4TB 10k rpm SAS HDDs, with an SSD as SLOG
  5. I have tested the system performance with fio, always using the following parameters:

    fio --ioengine=posixaio --bs=4k --directory=/mountpoint --refill_buffers --iodepth=1 --file_service_type=sequential --create_on_open=1 --fallocate=0 --unlink=1 --name=benchmark1 --rw=randwrite --numjobs=1 --nrfiles=500000 --filesize=20KB

Testing locally on the server showed a write speed of about 55MB/s.

Testing the same workload over NFS showed a measly performance of only 6000kB/s. The mount options on the client are as follows:

myserver:/export/path on /mnt/tmp type nfs4 (rw,relatime,vers=4.1,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=clientip,local_lock=none,addr=serverip)

I have also tested a sequential write of a single large file, to check for network limitations. For this workload I get 123MB/s, which is due to the NFS client being limited by gigabit ethernet.

I have also measured the performance after disabling sync writes in both NFS (vfs.nfsd.async=1) and ZFS (zfs set sync=disabled /myzpool/dataset). Without sync writes I get 9000kB/s instead of 6000, which is still unacceptable.

I do not have the option to use a MTU greater than 1500, because I do not have access to some of the routers/switches in the building.

1
  • edit the title removing "on FreeBSD system". Care more about NFS rather than what operating system it is.
    – ron
    Aug 3, 2023 at 14:41

2 Answers 2

1
  1. Increase NFS read and write buffer sizes
    sudo mount -o remount,rsize=262144,wsize=262144 myserver:/export/path /mnt/tmp
    
  2. Adjust the number of NFS threads
    sudo sysctl -w vfs.nfsd.maxthreads=256
    
  3. Optimize ZFS settings
    sudo zfs set recordsize=4K myzpool/dataset
    
0

Asked 3 years, 6 months ago

async and sync are nfs-server side export options, and are not relevant on the client side via mount. On the server side do an exportfs -s to observe whether it is either sync or async, and adjust that via /etc/exports. Your /etc/exports file could be as simple as this for example

/data *(rw,async)

If you edit that between sync and async and then do an exportfs -arv and then verify the change with exportfs -s then you will immediately observe the 2x speed difference over NFS. This is the most significant NFS speedup I have found. Be aware of data loss risk with async.

The rsize=1048576,wsize=1048576 on the client side shown by mount, I believe is the largest you can get it today, this is with NFS vers 4.1/4.2 and in RHEL 7.9 or 8.8. If anyone knows more let me know. It is set to the largest value by default (in RHEL at least) and then you would only reduce it because...

From https://www.ibm.com/docs/en/aix/7.2?topic=client-read-write-size-adjustments

Last Updated: 2023-03-24; for all products / AIX / 7.2 Reducing the rsize and wsize values might improve the NFS performance in a congested network by sending shorter packets for each NFS-read reply and write request. But, a side effect of this is that more packets are needed to send data across the network, increasing total network traffic, as well as CPU utilization on both the server and client. If your NFS file system is mounted across a high-speed network, such as Gigabit Ethernet, larger read and write packet sizes might enhance NFS file system performance. With NFS Version 3 and NFS Version 4, you can set the rsize and wsize values as high as 65536 when the network transport is TCP. The default value is 32768. With NFS Version 2, the maximum values for the rsize and wsize options is 8192, which is also the default.

I don't believe rsize and wsize is of much value today with gigabit and better networks. If you can scp test.tar which is a single 1gb or larger file and get a consistent 110 MB/sec or larger then network congestion is not a problem; For reference ~112MB/sec for a single file scp (secure copy) over SSH is the most you will see on a 1gigabit network. Slower speeds can be for a number of reasons not just network congestion, so make sure cpu and i/o load on your nfs servers are minimal.

My experience I have not been able to get NFS vers=4.2 working in RHEL 7.9, only vers=4.1 in RHEL 7.9 but did get vers=4.2 in RHEL 8.8. So you may experience that with your given linux operating system and with what version of NFS it comes with. Articles state significant improvements of nfs vers 4.2 versus 4.1, as well as nfs4 vs nfs3. I believe these imporovements are more for high traffic environments, not just getting greater MB/sec speed of a single file copy over NFS. Also, the 3 protocols to choose from are udp, tcp, and rdma. vers=rdma will be the best as it will save cpu cycles but you need network hardware {i.e. infiniband} to support that. I have not yet first hand tested to find if there is a single file NFS speed difference between vers=3 and vers=4.2 in RHEL 8.8. Nor if there is a difference between proto=tcp and proto=udp but in principle udp should be faster than udp. I have had trouble getting udp to be in effect in RHEL 8 to find out.

On the client side using fsc file system cache, as a mount option is a possible performance improver. Verify it by seeing fsc present as an option from doing mount. But for zero load and a single nfs copy speed test I do not think it will help.

I use rsync -P to copy a single file to get my MB/sec speed numbers.

This site is temporarily in read-only mode and not accepting new answers.

Not the answer you're looking for? Browse other questions tagged .