I've been trying to determine why/how my CentOS 7.3 NFS server hung, and I'm hoping to gain some insight or direction with this post. The machine's purpose is to serve NFS storage to clients. I use Nagios to monitor the server, and Ganglia/Graphite to collect and view metrics. I first noticed the issue when Nagios checks returned an unknown status and indeed I couldn't SSH to the server. I poked around a bit at the console but eventually gave up and rebooted the machine as clients started complaining. The reboot seemed to resolve the issue.
The logs really only had a couple events of interest: a good number of 'nfsd' warnings followed by a flood of 'file-max limit <> reached' errors:
- nfsd: too many open connections, consider increasing the number of threads
- VFS: file-max limit 6509328 reached
The Ganglia metrics showed a spike in NFS operations and connections ~1 hour prior the file-max limit errors. The metrics also showed the number of open files increasing steadily over that hour (normally, the number of open files is ~7K with the largest spikes being ~100K).
Here are the questions I'm struggling with:
I don't have much visibility of the NFS clients and I'm suspicious that their workloads may have triggered the issue. The number of NFS threads (RPCNFSDCOUNT) is set to 24 and 'ulimit -n' returns 1024. Does this mean the maximum number of open files the NFS clients could be using is 24 x 1024?
While the system was reporting that the file-max limit was reached, I was unable to identify the process responsible. Looking at the output of lsof, the two leading processes were /usr/sbin/gssproxy and /usr/sbin/tuned, but neither reported that many open files (a couple hundred each IIRC). Researching the file-max limit issue, I found some mention of bugs that resulted in file descriptor leaks but nothing that was relevant to the versions I'm running.
Is it possible this is the result of some aggressive client(s) or does it seem more likely to be a leak in one of the processes (uptime was about 300 days). Either way, are there any suggestions for how to identify the root cause?