How can I resolve autofs mount delays related to host probing?

Question

I have an infrastructure of CentOS 6 and CentOS 7 client machines that rely on autofs to automount various NFS filesystems exported by a service elsewhere in my organization. Recently, the clients began manifesting a troublesome behavior in which automounting these filesystems became very slow -- whereas mounting used to go through in a few seconds, it began to take nearly two minutes.

I think I have traced down the problem to a combination of factors:

• The hostname of the server has a large number of distinct resolutions (32)
• When the hostname has multiple resolutions, autofs probes each one to try to reject unresponsive ones and choose the one among the rest that currently has the best response time
• Exactly one of the two probe RPCs issued to each server by autofs appears to be consistently timing out for all of my servers.

Here's a representative excerpt of the debug log:

Jul 13 15:48:18 myclient automount[17485]: get_nfs_info: called with host nfs.my.org(10.220.8.68) proto 6 version 0x20
Jul 13 15:48:18 myclient automount[17485]: get_nfs_info: nfs v3 rpc ping time: 0.000290
Jul 13 15:48:18 myclient automount[17485]: get_nfs_info: host nfs.my.org cost 289 weight 0
Jul 13 15:48:18 myclient automount[17485]: get_nfs_info: called with host nfs.my.org(10.220.8.68) proto 17 version 0x20
Jul 13 15:48:21 myclient automount[17485]: get_nfs_info: called with host nfs.my.org(10.220.8.84) proto 6 version 0x20

That shows one complete probe and the beginning, three seconds later, of the following one. In addition to the delay, I don't see any information about a response to the second RPC. That says "timeout" to me. Although the timeouts are individually only 3 seconds, multiplying that by 32 machines means over a minute and a half of timeout before the mount itself is actually attempted.

The clients are running the standard NFS client stacks for CentOS 6 and 7: nfs-utils 1.2.3 and autofs 5.0.5 or nfs-utils 1.3.0 and autofs 5.0.7, respectively, as packaged by CentOS. Clients are under configuration management, so I am confident that they have had no software or configuration change since well before the problem began manifesting.

The servers are running the Ganesha userspace NFS stack, and in particular, it may be relevant that they do not support NFS4, though this has not presented a problem in the past. Server management claims that no configuration change has been intentionally made, but allows that routine software updates may have been installed.

So, finally, the question is as given in the headline: how can I resolve the mount delays caused by the host probing? Is there a relevant configuration setting in Ganesha whose default may have changed? Alternatively, is there a way to configure autofs to avoid trying the failing RPCs? Or have I perhaps mis-identified the problem?

Turning on the autofs config parameter use_hostname_for_mounts seems to work around the issue, but as I understand it, this comes at the cost of losing resilience against failures and overloading of the individual servers. Is there no better way?

Answer 1

It seems that that the key clue in the log messages is that the probes logged as having "proto 6" succeed and those logged as having "proto 17" fail. The 6 and 17 turn out to be the IP transport protocol numbers representing TCP and UDP, respectively.

Although NFS is traditionally served over UDP, service over TCP is supported by most stacks, and the server in this case was always configured to serve NFS only over TCP. This did not present a problem, however, until an as-yet uncharacterized change went in at the server that had the result that nfs/udp traffic was afterward silently discarded instead of being rejected with the appropriate ICMP response. That might very well have arisen from a firewall change, but I cannot at this point rule out an application-level change at the server.

In any event, I resolved the problem on the client side, by adding proto=tcp to the mount options of each affected filesystem in the autofs map file. Autofs was clever enough to forgo the UDP-flavor probes once that option was in place. Not only is the problem solved, but mount performance now seems even a little better than it was before the timeout problem started.