56

Am doing some work on a remote CentOS 5.6 machine and my network keeps dropping. Is there a way that I can recover my hung sessions after I reconnect?

EDIT: am doing some updating and installing with yum and am worried this might be a problem if processes keep hanging in the middle of whatever they're doing.

Improve this question
1
  • 8
    tmux or screen.
    – sehe
    Aug 23, 2011 at 9:40

7 Answers 7

Reset to default
86

There is no way, but to prevent this I like using tmux. I start tmux, start the operation and go on my way. If I return and find the connection has been broken, all I have to do is reconnect and type tmux attach.

Here's an example.

$ tmux
$ make <something big>
......
Connection fails for some reason
Reconect

$ tmux ls
0: 1 windows (created Tue Aug 23 12:39:52 2011) [103x30]

$ tmux attach -t 0
Back in the tmux sesion
Improve this answer
3
  • 1
    dont have tmux, and am not allowed to install things not on my to do list...
    – sergio
    Aug 23, 2011 at 9:44
  • 7
    @sergio My heart bleeds :-)) Use screen.
    – cnicutar
    Aug 23, 2011 at 9:45
  • 6
    @sergio On a Unix system, virtually any application that does not require root permissions to run may be installed in ones home directory.
    – Kusalananda ♦
    Apr 14, 2017 at 7:10
37

The recommendations for tmux and screen are both good suggestions. They imply the answer to your question, but don't actually state it. The answer to the question is: there is no way. If you login via ssh, the shell is ended when the connection drops. The work-around is to login and immediately start a virtual terminal of some kind (this can be done with tools like dtach, abduco, tmux, or screen). When the connection drops, the shell you are in is ended, but you can open a new shell and reconnect to the virtual terminal that is running the shell in which you are actually doing your work.

Improve this answer
4
  • 1
    Assuming the yum process still runs (didn't immediately terminate when the shell got SIGHUP), reptyr or similar might be sufficient to recover the process or --failing that-- obtain any future output. The shell would generally be terminated on disconnect, though.
    – Eroen
    Dec 7, 2012 at 22:28
  • @Eroen You mean, even whe using tmux, the OS will terminate the tmux process once it finds that the connection is lost?
    – Dojo
    Oct 23, 2015 at 12:26
  • @Dojo When the connection is terminated, the tmux instance will be stopped, but the tmux session (and the shells it manages) will stay up.
    – William Pursell
    Oct 23, 2015 at 15:53
  • If the user lost the connection to the remote ssh server because for example he suddenly accidentally unplugged the RJ-45 cable, I think the ssh server still keeps this session until a predefined timeout (120 seconds for example). So in this case, is there a way to resume this session that is still alive on the ssh server side within 120 seconds after the cable is unplugged?
    – Gab是好人
    Sep 16, 2016 at 14:03
9

As William said, the short answer is no, there is no way. To prevent this you could use the screen command before you lose connection

Improve this answer
4
  • from what i understand, i would have to have initiated yum with screen...welll, i didnt. i cant rerun yum it says its still being used and i dont want to force kill it. ..how can i regain control of the running yum?
    – sergio
    Aug 23, 2011 at 9:41
  • and byobu is a nice complement to screen , to get it automatically launched in a nice graphical terminal launchpad.net/byobu
    – regilero
    Aug 23, 2011 at 9:43
  • As far as I know you can't, unless there's some hidden way to redirecting the output from the yum command into your current term session but I can't think of one off the top of my head.
    – Nicholas Smith
    Aug 23, 2011 at 9:44
  • There are a number of partial solutions to this using debuggers & c. to give a process a new parent shell. Reptyr is one, and a blog post describes the issue, the workaround and some other implementations.
    – Eroen
    Dec 7, 2012 at 22:23
9

Someone added reptyr in a comment rather than an answer so I can't upvote it, but it seems a good answer to the edited question and worked great on CentOS.

see: https://github.com/nelhage/reptyr https://www.ostechnix.com/reptyr-move-running-process-new-terminal/

Is there a way that I can recover my hung sessions after I reconnect?

EDIT: am doing some updating and installing with yum and am worried this might be a problem if processes keep hanging in the middle of whatever they're doing.

So same thing, I lost remote connection to a yum update session. So I searched and found this question... then reconnected and used screen this time:

  1. reconnect ssh
  2. screen
  3. install reptyr as per the links above
  4. ps -a | grep yum
  5. reptyr psid

and there I am at the recovered prompt which yum was giving some hours earlier.

Technically the answers are all correct, this is NOT a recovery of a hung session, it is reparenting of an orphaned process to the current session. And very useful it is too...

Improve this answer
1
  • This worked to recover my ddrescue. I don't see why everybody is saying it's impossible when this 2011 program can do it just fine.
    – user
    Apr 14, 2022 at 13:23
3

No, you can't recover a shell after a disconnection. Instead what you can do is ensure the command you were running continues to run after the disconnect.

To achieve this, use "nohup" and "disown" commands, which are usually builtin commands on most shells, i.e. you don't need to install anything. This only works for non-interactive commands though.

So, the steps would be as follows:

  1. Login to server
  2. Run your command: "nohup sudo yum update &" (this will also log all output to nohup.out file in your current directory)
  3. Run "disown %1"
  4. Feel free to disconnect at this point or wait to be disconnected :)

When you come back to the server, simply "tail nohup.out" to see how the command is doing.

Improve this answer
3
  • Wouldn't this work pretty poorly in the general case? For example, with commands that want input at one time or another... For completely non-interactive commands, it might work well enough and avoid the overhead of (and possibly needing to install) a terminal multiplexer.
    – user
    Sep 17, 2013 at 14:11
  • Yes, this only works if your commands don't want any user input, I've updated the answer. Overall a terminal multiplexer is the way to go for sure, but it was mentioned that no additional tools were permitted to be installed.
    – zygis
    Sep 17, 2013 at 15:03
  • Absolutely, it's a valid answer, just slightly limited in its possible use cases. Upvoted with the edit.
    – user
    Sep 18, 2013 at 7:29
1

As many people have suggested screen and tmux, they both support basic functionality, but have distinct specific features, so one can't say that one is superior to another in all cases. For example, only tmux supports Window-splitting, while only GNU screen can toggle long line wrapping with (Ctrl+a r). See more detailed comparison here.

There also exist tools specifically to fix this problem of ssh:

Autossh is a program to start a copy of ssh and monitor it, restarting it as necessary should it die or stop passing traffic. The idea is from rstunnel.

Mosh is a remote terminal application that allows roaming, supports intermittent connectivity, and provides intelligent local echo and line editing of user keystrokes. Mosh is a replacement for SSH. It's more robust and responsive, especially over Wi-Fi, cellular, and long-distance links.

Improve this answer
-1

This can be done using tmux shell. If you get disconnected the fastest way to reattach a session is:

tmux a #0
Improve this answer
2
  • This adds nothing that the accepted answer does not say. It also assumes that the user already has a tmux session running.
    – Kusalananda ♦
    Feb 20, 2019 at 17:19
  • 2
    This could be a comment on that answer saying "you can abbreviate attach as a".
    – Jeff Schaller ♦
    Feb 20, 2019 at 17:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy