19

Summary: I'm trying to figure out why my tmux session dies when I disconnect from ssh

Details:

I have tmux installed on an Arch Linux system. When I start a tmux session I can detach from it and then attach again while the ssh session is active. But if I end my ssh session then the tmux session gets killed.

I know this is not the normal behavior because I have other system where the tmux session continues running even if the ssh session is ended and I can attach to the tmux session after establishing a new ssh connection. The system that has a problem and the one that works correctly have very similar configurations so I'm not sure what to check.

I'm running tmux version 1.9a. The system that has a problem (that I have root access for) has a Linux kernel version of 3.17.4-1 and the system that works correct has kernel version 3.16.4-1-ARCH (I don't have root on that system). I doubt that the kernel version is the source of the problem though, that's just one difference I noticed.

I thought I'd ask to see if anyone has seen a similar problem and knows of a possible solution.

The precise steps that lead to the problem are:

  1. ssh to machine
  2. run tmux to start tmux
  3. ctrl-B D to detach (at this point I could reattach with tmux attach
  4. close ssh session (at this point the tmux session is killed, I've been able to observe this when I'm logged in as root in a different terminal)
  5. reconnect with ssh and run tmux attach and I get the message no sessions and running tmux ls returns failed to connect to server: Connection refused. This makes sense because the serve is not running. What doesn't make sense to me is why it gets killed in step 4 when I disconnect from the ssh session.

strace data:

In response to one of the comments I used strace to see what systems calls the tmux server process makes. It looks like when I exit my ssh session (by typing exit or with ctrl-d) that the tmux process is being killed. Here's a snippet of the final part of the strace output.

poll([{fd=4, events=POLLIN}, {fd=11, events=POLLIN}, {fd=6, events=POLLIN}], 3, 424) = ? ERESTART_RESTARTBLOCK (Interrupted by signal)
--- SIGTERM {si_signo=SIGTERM, si_code=SI_USER, si_pid=1, si_uid=0} ---
sendto(3, "\17", 1, 0, NULL, 0)         = 1
+++ killed by SIGKILL +++

I compared this with a different system where tmux works properly and on that system the tmux process continues running even after I exit. So the root cause appears to be that the tmux process is being terminated when I close the ssh session. I'll need to spend some time troubleshooting this to figure out why, but I thought I would update my question since the strace suggestion was useful.

migrated from serverfault.com Dec 4 '14 at 16:34

This question came from our site for system and network administrators.

  • to be sure, please describe step by step : I assume you ssh, start a tmux session, detach from the session, and close the shh : when you ssh again, you don t have any way to re join the tmix session? ie, the session is no longer running? – Olivier Dulac Dec 4 '14 at 18:18
  • @OlivierDulac yes your assumption is correct. I've also edited my question to include these details. – Gabriel Southern Dec 4 '14 at 18:47
  • how do you close the ssh session? and you could attach a strace to the pid of tmux and another to the pid of the sshd, to see if it receives something when you close the ssh connection (very verbose, redirect to a file) – Olivier Dulac Dec 5 '14 at 8:53
  • @OlivierDulac thanks for the suggestion. I've updated the question with info from strace. It looks like the tmux server process is getting killed when I end the ssh session. I don't think this is supposed to happen, so I need to figure out why it is happening. – Gabriel Southern Dec 5 '14 at 22:52
  • Start tmux with verbose logging enabled and see if anything is printed to the log when you disconnect. Also, what is the TERM on the remote machine in and out of tmux? – jasonwryan Dec 5 '14 at 23:08
13
+50

Theory

Some init systems including systemd provide a feature to kill all processes belonging to the service. The service typically starts a single process which that creates more processes by forking and those processes can do that as well. All such processes are typically considered part of the service. In systemd this is done using cgroups.

In systemd, all processes belonging to a service are killed when the service is stopped by default. The SSH server is obviously part of the service. When you connect to the server, SSH server typically forks and the new process handles your SSH session. By forking from the SSH session process or its children, other server side processes are started, including your screen or tmux.

Killmode and socket activation

The default behavior can be changed using the KillMode directive. The upstream project doesn't AFAIK include any .service files and so those vary by distribution. There are typically two ways to enable SSH on your system. One is the classic ssh.service that maintains a long running SSH daemon listening on the network. The other is via socket activation handled by the ssh.socket that in turn starts sshd@.service which only runs for a single SSH session.

Solutions

If your processes get killed at the end of the session, it is possible that you are using socket activation and it gets killed by systemd when it notices that the SSH session process exited. In that case there are two solutions. One is to avoid using socket activation by using ssh.service instead of ssh.socket. The other is to set KillMode=process in the Service section of ssh@.service.

The KillMode=process setting may also be useful with the classic ssh.service, as it avoids killing the SSH session process or the screen or tmux processes when the server gets stopped or restarted.

Future notes

This answer apparently gained a level of popularity. While it worked for the OP it might happen that it doesn't work for someone in the future due to systemd-logind development or configuration. Please check documentation on logind sessions if you experience behavior different from the description in this answer.

  • Any specific feedback from the downvoter or just trolling? – Pavel Šimerda Dec 30 '14 at 7:50
  • 3
    Thanks for the detailed response. Switching to sshd.service fixed the problem. – Gabriel Southern Jan 5 '15 at 17:41
  • I experience this problem on a system using init rather than systemd. But it's slightly different anyway, see my question. – gerrit Nov 14 '16 at 20:47
4

Do you use systemd with socket activation for SSH?

If so, there’s a known issue with that. According to the systemd proponents, this is actually a feature – systemd kills all processes spawned by a session when the session terminates. (I can see that being useful, but in the GNU screen, or tmux, case, you definitely don’t want that ☺ nor in most other cases where users may run background processes, of course.)

If so, try switching from sshd.socket to sshd.service.

  • 1
    I would say you generally don't want to use that feature for SSH logins if your users are allowed to run processes that are running after logging out. That's not specific to screen or tmux but rather to SSH (with any background processes on the server side). – Pavel Šimerda Dec 25 '14 at 9:14
  • 2
    @PavelŠimerda yes, I thought that implicit, but edited the post to make it more explicit now. – mirabilos Dec 28 '14 at 12:54
2

Another solution to this, which doesn't require moving from sshd.socket to sshd.service, is to start tmux server as a systemd service [0]. This way, the tmux server is already running when you SSH into the server, instead of spawned by the tmux command in SSH, thus won't be killed.

[0] https://wiki.archlinux.org/index.php/tmux#Autostart_with_systemd

  • Does this work when I have not got root? – gerrit Nov 14 '16 at 20:40
  • Yeah, that's a valid solution. But you still want to solve the case with restarting SSH service over SSH session. :) – Pavel Šimerda Nov 15 '16 at 15:26
2

I was having the same problem with tmux and screen on Ubuntu 16.04 (kde neon). When the ssh session was disconnected screen / tmux was terminated.

long story short, systemd changed their default setting to killuserprocess=yes so after leaving a ssh session every process created by it will be terminated.

Easy fix (after hours of trying) run screen/tmux using this command

For screen

systemd-run --scope --user screen

for Tmux

systemd-run --scope --user tmux

You can create an alias to make it easier

alias tmux= "systemd-run --scope --user tmux"

  • -bash: systemd-run: command not found on Red Hat Enterprise Linux Server release 6.8 (Santiago). – gerrit Nov 14 '16 at 19:49
  • Does this work when I have not got root? – gerrit Nov 14 '16 at 20:42
  • 1
    I noticed the undesired tmux/screen killing behavior does not happen on Ubuntu 18.04 LTS, only 16.04. – Seth Aug 4 '18 at 17:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.