On my work computer, I regularly SSH into two unrelated servers. Sometimes the SSH session locks up (does not respond to keyboard input). When locked up, I can SSH in via a second open terminal, that works but does not unfreeze the frozen terminal. Pressing ~. takes some time before the terminal disconnects and gives me a local CLI. I regularly SSH into both those servers from home with no problems, so I suspect that the issue is on my local work computer.

Does the problem occur with other terminals? Yes, the problem occurs with both Konqueror and with Terminator.

Does the problem occur to both remote servers at the same time? No

Does the problem occur with any particular application running on the server? Both with and without GNU Screen, both on the CLI and in VIM.

To test, I just opened four terminals: two Konqueror terminals, one to each server; and two Terminator terminals, one to each server. After about an hour, only one (Terminator) terminal locked up. The other terminal to the same server did not lock up, and neither terminal to the other server locked up. Of course, I tried Ctrl-Q in the locked up terminal to see if an errant Ctrl-S had been sent, but that did not resolve the issue. Screen was not running in the terminal that locked up (neither in the native session that is running the SSH, nor the SSH session itself).

UPDATE: An hour later the server locked up in Terminator has finally disconnected with "Write failed: Broken pipe" and I have a working local CLI, and the two servers in Konsole are also locked up.

  • That's just the connection; mine does the same thing if I leave an ssh session up and put the machine into sleep for a while. When it wakes up, the connection is of course defunct, but ssh is frozen. If I ignore it for long enough it may die properly, usually I just kill the terminal emulator.
    – goldilocks
    Jan 14 '13 at 13:09

Something along the chain is timing out the idle connection, since SSH doesn't normally send anything when idle. But, you can make it send messages periodically when idle. In OpenSSH version 3.8 and up:

$ ssh -oServerAliveInterval=60 myremotebox

If you're going to ssh manually to this host frequently, you probably want to put it in your ~/.ssh/config file instead:

Host myremotebox

This tells it to send a null packet every 60 seconds after nothing else has been sent. I've found across a wide variety of infrastructure that this is enough to keep the connection alive.

In pre-3.8 versions of OpenSSH, you don't have this option, but there is a weak fallback. You can set the KeepAlive option, which uses TCP keepalives. The way this works is OS-dependent, and often changing its behavior affects all applications. Worse, network stacks typically default to sending TCP keepalives every 2 hours by default, so you almost have to change the default if you're going to use it this way, since the thing timing out your SSH connection probably has an idle threshold much lower than 2 hours.

Please note, if you're reading version 3.8+ docs, that this is the same thing as the TCPKeepAlive option. When they added the "server alive" option in 3.8, they renamed KeepAlive to TCPKeepAlive to distinguish the two.

  • You would think the client should have the brains to know when it is defunct tho. Belongs on a TODO somewhere :/
    – goldilocks
    Jan 14 '13 at 13:29
  • You've never written a network client program, have you? :) The only way you can detect such a situation is to send keepalive packets; it's why TCP has this feature in the first place. Ideally, the thing timing out the connection would send a TCP RST to both ends, but a lot of infrastructure doesn't do this. It simply ceases to pay attention to packets on that connection. This is typical of NAT routers. Jan 14 '13 at 13:30
  • Point taken -- I have actually written minor network clients and I suppose they are subject to the same issue; TCP sockets do not always report as closed and may simply block/hang if the other end does not disconnect properly. I've also (toot, toot) written an http server and it needs to time out all connections to account for abherrant clients, which AFAIK is a feature common to all http (and probably tcp) servers.
    – goldilocks
    Jan 14 '13 at 13:51
  • 1
    @goldilocks: When your HTTP server times out connections, it calls close(2) on the socket, which causes either a TCP FIN or a TCP RST, depending on how you do this. (Toot, toot, too.) When, say, a NAT router simply stops paying attention to a forwarded connection, no packets go to each end, so until one of the peers tries sending data again, it's not going to get any notification that its connection no longer works. This is considered a feature, since things that break in networks do sometimes get fixed before anyone cares. Jan 14 '13 at 13:54
  • Okay, so the router is working a layer below tcp and if it stops forwarding, neither end will be properly notified and the sockets remain open; if you kill a process arbitrarily on one end that can happen too. In this case a server, which is either threaded, forked or async/non-block, can time out a connection that doesn't respond to IO, but a singular client may not have this capability. It could though -- but I can understand why one might not bother to implement it that way (more trouble that it's worth).
    – goldilocks
    Jan 14 '13 at 14:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.