I spent quite some time tracking down a problem in production recently, where a database server disappearing would cause a hang of up to 2 hours (long wait for a
poll() call in the libpq client library) for a connected client. Digging into the problem, I realized that these kernel parameters should be adjusted way down in order for severed TCP connections to be noticed in a timely fashion:
net.ipv4.tcp_keepalive_time = 7200
net.ipv4.tcp_keepalive_probes = 9
net.ipv4.tcp_keepalive_intvl = 75
net.ipv4.tcp_retries2 = 15
The four values above are from an Ubuntu 12.04 machine, and it looks like these defaults are unchanged from current Linux kernel defaults.
These settings seem to be heavily biased towards keeping an existing connection open, and being extremely stingy with keepalive probes. AIUI, the default
tcp_keepalive_time of 2 hours means when we're waiting for a response for a remote host, we will wait patiently for 2 hours before initiating a keepalive probe to verify our connection is still valid. And then, if the remote host does not respond to a keepalive probe, we retry those keepalive probes 9 times (
tcp_keepalive_probes), spaced 75 seconds apart (
tcp_keepalive_intvl), so that's an extra 11 minutes before we decide the connection is really dead.
This matches what I've seen in the field: for example, if I start a
psql session connected to a remote PostgreSQL instance, with some query waiting on a response, e.g.
and then have the remote server die a horrible death (e.g. drop traffic to that machine), I see my psql session waiting for up to 2 hours and 11 minutes before it figures out its connection is dead. As you might imagine, these default settings cause serious problems for code which we have talking to a database during, say, a database failover event. Turning these knobs down has helped a lot! And I see that I'm not alone in recommending these defaults be adjusted.
So my questions are:
- How long have the defaults been like this?
- What was the original rationale for making these TCP settings the default?
- Do any Linux distros change these default values?
And any other history or perspective on the rationale for these settings would be appreciated.