I have an infrastructure of CentOS 6 and CentOS 7 client machines that rely on autofs to automount various NFS filesystems exported by a service elsewhere in my organization. Recently, the clients began manifesting a troublesome behavior in which automounting these filesystems became very slow -- whereas mounting used to go through in a few seconds, it began to take nearly two minutes.
I think I have traced down the problem to a combination of factors:
- The hostname of the server has a large number of distinct resolutions (32)
- When the hostname has multiple resolutions, autofs probes each one to try to reject unresponsive ones and choose the one among the rest that currently has the best response time
- Exactly one of the two probe RPCs issued to each server by autofs appears to be consistently timing out for all of my servers.
Here's a representative excerpt of the debug log:
Jul 13 15:48:18 myclient automount: get_nfs_info: called with host nfs.my.org(10.220.8.68) proto 6 version 0x20 Jul 13 15:48:18 myclient automount: get_nfs_info: nfs v3 rpc ping time: 0.000290 Jul 13 15:48:18 myclient automount: get_nfs_info: host nfs.my.org cost 289 weight 0 Jul 13 15:48:18 myclient automount: get_nfs_info: called with host nfs.my.org(10.220.8.68) proto 17 version 0x20 Jul 13 15:48:21 myclient automount: get_nfs_info: called with host nfs.my.org(10.220.8.84) proto 6 version 0x20
That shows one complete probe and the beginning, three seconds later, of the following one. In addition to the delay, I don't see any information about a response to the second RPC. That says "timeout" to me. Although the timeouts are individually only 3 seconds, multiplying that by 32 machines means over a minute and a half of timeout before the mount itself is actually attempted.
The clients are running the standard NFS client stacks for CentOS 6 and 7: nfs-utils 1.2.3 and autofs 5.0.5 or nfs-utils 1.3.0 and autofs 5.0.7, respectively, as packaged by CentOS. Clients are under configuration management, so I am confident that they have had no software or configuration change since well before the problem began manifesting.
The servers are running the Ganesha userspace NFS stack, and in particular, it may be relevant that they do not support NFS4, though this has not presented a problem in the past. Server management claims that no configuration change has been intentionally made, but allows that routine software updates may have been installed.
So, finally, the question is as given in the headline: how can I resolve the mount delays caused by the host probing? Is there a relevant configuration setting in Ganesha whose default may have changed? Alternatively, is there a way to configure autofs to avoid trying the failing RPCs? Or have I perhaps mis-identified the problem?
Turning on the autofs config parameter
use_hostname_for_mounts seems to work around the issue, but as I understand it, this comes at the cost of losing resilience against failures and overloading of the individual servers. Is there no better way?