I have a FreeBSD storage Server exporting ZFS Datasets via NFS, but the performance for small file transfers is below the acceptable limit.

Some background info:

  1. The exports are handeled via /etc/exports, not by the sharenfs property in ZFS.
  2. The Server has 128GB RAM and 2x8 core CPU's
  3. The NFS server is configured to use 192 threads (vfs.nfsd.maxthreads: 192)
  4. The zpool spans 4x4TB 10k rpm SAS HDDs, with an SSD as SLOG
  5. I have tested the system performance with fio, always using the following parameters:

    fio --ioengine=posixaio --bs=4k --directory=/mountpoint --refill_buffers --iodepth=1 --file_service_type=sequential --create_on_open=1 --fallocate=0 --unlink=1 --name=benchmark1 --rw=randwrite --numjobs=1 --nrfiles=500000 --filesize=20KB

Testing locally on the server showed a write speed of about 55MB/s.

Testing the same workload over NFS showed a measly performance of only 6000kB/s. The mount options on the client are as follows:

myserver:/export/path on /mnt/tmp type nfs4 (rw,relatime,vers=4.1,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=clientip,local_lock=none,addr=serverip)

I have also tested a sequential write of a single large file, to check for network limitations. For this workload I get 123MB/s, which is due to the NFS client being limited by gigabit ethernet.

I have also measured the performance after disabling sync writes in both NFS (vfs.nfsd.async=1) and ZFS (zfs set sync=disabled /myzpool/dataset). Without sync writes I get 9000kB/s instead of 6000, which is still unacceptable.

I do not have the option to use a MTU greater than 1500, because I do not have access to some of the routers/switches in the building.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.