I spent quite some time tracking down a problem in production recently, where a database server disappearing would cause a hang of up to 2 hours (long wait for a poll() call in the libpq client library) for a connected client. Digging into the problem, I realized that these kernel parameters should be adjusted way down in order for severed TCP connections to be noticed in a timely fashion:

net.ipv4.tcp_keepalive_time = 7200 net.ipv4.tcp_keepalive_probes = 9 net.ipv4.tcp_keepalive_intvl = 75 net.ipv4.tcp_retries2 = 15

The four values above are from an Ubuntu 12.04 machine, and it looks like these defaults are unchanged from current Linux kernel defaults.

These settings seem to be heavily biased towards keeping an existing connection open, and being extremely stingy with keepalive probes. AIUI, the default tcp_keepalive_time of 2 hours means when we're waiting for a response for a remote host, we will wait patiently for 2 hours before initiating a keepalive probe to verify our connection is still valid. And then, if the remote host does not respond to a keepalive probe, we retry those keepalive probes 9 times (tcp_keepalive_probes), spaced 75 seconds apart (tcp_keepalive_intvl), so that's an extra 11 minutes before we decide the connection is really dead.

This matches what I've seen in the field: for example, if I start a psql session connected to a remote PostgreSQL instance, with some query waiting on a response, e.g.

SELECT pg_sleep(30);

and then have the remote server die a horrible death (e.g. drop traffic to that machine), I see my psql session waiting for up to 2 hours and 11 minutes before it figures out its connection is dead. As you might imagine, these default settings cause serious problems for code which we have talking to a database during, say, a database failover event. Turning these knobs down has helped a lot! And I see that I'm not alone in recommending these defaults be adjusted.

So my questions are:

  • How long have the defaults been like this?
  • What was the original rationale for making these TCP settings the default?
  • Do any Linux distros change these default values?

And any other history or perspective on the rationale for these settings would be appreciated.

  • Some relevant info here... tldp.org/HOWTO/TCP-Keepalive-HOWTO/usingkeepalive.html – Drav Sloan Aug 11 '15 at 14:41
  • Note that you can change the first three per-connection in client code with the socket options TCP_KEEPIDLE, TCP_KEEPCNT, and TCP_KEEPINTVL. – wnoise Aug 18 '15 at 21:16
  • 1
    @wnoise actually since Linux 2.6.37 it should also be possible to specify socket option TCP_USER_TIMEOUT as well, instead of setting net.ipv4.tcp_retries2 system-wide. Of course many applications (such as PostgreSQL in my example here) don't support TCP_USER_TIMEOUT yet. – Josh Kupershmidt Aug 19 '15 at 14:27
up vote 6 down vote accepted
+50

RFC 1122 specifies in section 4.2.3.6 that the keep-alive period must not default to less than two hours.

  • 1
    Nice, thank you for digging that up. I think that mostly answers the question of why tcp_keepalive_time defaults to 7200, though I'd still be interested in the precedent/explanation for the the other three relevant settings. – Josh Kupershmidt Aug 18 '15 at 19:20
  • Removing My answer as this answers the question (at least for one of the values) – coteyr Aug 18 '15 at 19:48
  • 1
    @coteyr Thanks anyway, I appreciate the effort. IIRC there was an intriguing comment on your answer suggesting that in earlier Linux kernels the default was 15 minutes. I'd be interested in how/why that got changed to 2 hours, or set to 15 minutes in the first place. – Josh Kupershmidt Aug 19 '15 at 1:00

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.