I have a FreeBSD storage Server exporting ZFS Datasets via NFS, but the performance for small file transfers is below the acceptable limit.
Some background info:
- The exports are handeled via /etc/exports, not by the sharenfs property in ZFS.
- The Server has 128GB RAM and 2x8 core CPU's
- The NFS server is configured to use 192 threads (vfs.nfsd.maxthreads: 192)
- The zpool spans 4x4TB 10k rpm SAS HDDs, with an SSD as SLOG
I have tested the system performance with fio, always using the following parameters:
fio --ioengine=posixaio --bs=4k --directory=/mountpoint --refill_buffers --iodepth=1 --file_service_type=sequential --create_on_open=1 --fallocate=0 --unlink=1 --name=benchmark1 --rw=randwrite --numjobs=1 --nrfiles=500000 --filesize=20KB
Testing locally on the server showed a write speed of about 55MB/s.
Testing the same workload over NFS showed a measly performance of only 6000kB/s. The mount options on the client are as follows:
myserver:/export/path on /mnt/tmp type nfs4 (rw,relatime,vers=4.1,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=clientip,local_lock=none,addr=serverip)
I have also tested a sequential write of a single large file, to check for network limitations. For this workload I get 123MB/s, which is due to the NFS client being limited by gigabit ethernet.
I have also measured the performance after disabling sync writes in both NFS (vfs.nfsd.async=1) and ZFS (zfs set sync=disabled /myzpool/dataset). Without sync writes I get 9000kB/s instead of 6000, which is still unacceptable.
I do not have the option to use a MTU greater than 1500, because I do not have access to some of the routers/switches in the building.