I sometimes have long running processes that I want to kick off before going home, so I create a SSH session to the server to start the process, but then I want to close my laptop and go home and later, after dinner, I want to check on the process that I started before leaving work. How can I do that with SSH? My understanding is that if you break your SSH connection you will also break your login session on the server, therefore killing the long running process.
You want to be using GNU Screen. It is super awesome!
ssh email@example.com screen #start a screen session run-a-long-process
CTRL+a , d to detatch from your screen session
exit #disconnect from the server, while run-a-long-process continues
When you come back to your laptop:
ssh firstname.lastname@example.org screen -r #resume the screen session
Then check out the progress of your long-running process!
screen is a very comprehensive tool, and can do a lot more than what I've described. While in a screen session, try ctrl+a,? to learn a few common commands. Probably the most common are:
- CTRL+a , c to create a new window
- CTRL+a , n to switch to the next window in your screen session
- CTRL+a , p to switch to the previous window in your screen session
- if you log in from a bunch of different systems, you may have accidentally left yourself attached to an active screen session on a different computer. for that reason, I always resume with
screen -d -rto ensure that if another shell is attached to my screen session, it will be detached before I resume it on my current system.
If you haven't planned ahead and setup
screen, etc. just do the following:
If your process is running in the background: goto #3, else:
Ctrl-Zto suspend foreground process. This will report the job # of the suspended process, for example:
+ Stopped processName
processNameto the background with
bg %1(using whatever the job # is following the
%). This will resume
processNamein the background.
disown PID. Use the
-hflag if you want to maintain ownership until you terminate your current shell.
What you want to use is screen or even better a user-friendly wrapper around screen called byobu.
byobu is a wrapper that allows to easily open new screens with a simple function key instead of key combination from ctrl-a. It also shows a status line with all the open virtual terminals which can be named.
Another nice feature is the fact that all your screen can stay up while your ssh connection is disconnected. You just connect again via ssh and call byobu and everything is like before.
At last some screenshots of byobu.
It might be worth noting that
ssh -t lala screen -rxU moo will attach to the moo session on host lala
ssh -t lala screen -S moo will create the moo session on host lala
ssh -t lala screen -S moo quux will create the moo session on host lala and run the program quux, quitting the session on completion.
Old question, strange still nobody advised tmux, which acts as a wrapper for n consoles and keep them open until needed. This allows more control, beside a number of functions tmux has. It's easy to manage it, you just execute tmux, which brings you in its shell, start your looong job, then press ctrl+b followed by d (detach) (
ctrl+b is the "ok google" of tmux, and
d is the command to close without exit from the shell). This actually works if you just close, for example, putty. After dinner, when you connect again, you can reopen tmux with
tmux attach to see your screen exactly as you left. Something I love is splitting pane:
ctrl+b and then press
". To change from one pane to another,
ctrl+b and then press the up/down arrow.
You can find a good guide here: Keep Your SSH Session Running when You Disconnect
sudo apt-get install screen
Now you can start a new screen session by just typing screen at the command line. You’ll be shown some information about screen. Hit enter, and you’ll be at a normal prompt.
To disconnect (but leave the session running) Hit Ctrl + A and then Ctrl + D in immediate succession. You will see the message [detached]
To reconnect to an already running session
To reconnect to an existing session, or create a new one if none exists
screen -D -r
To create a new window inside of a running screen session Hit Ctrl + A and then C in immediate succession. You will see a new prompt.
To switch from one screen window to another Hit Ctrl + A and then Ctrl + A in immediate succession.
To list open screen windows Hit Ctrl + A and then W in immediate succession
Tmux is a good option available to run your long-running processes in the background.
I have to keep running the long-running processes on a google cloud platform's VM instance/server (with OS: Ubuntu 16.0). Where I have to start SSH terminal and from the terminal, I have to keep the terminal connected to run the process. Now up to this, all is good. But wait if the connection with my SSH terminal is terminated then the long-running processes stop immediately and hence I have to re-run them once again once the ssh terminal is restarted or from a new ssh terminal.
tmux is the good solution to avoid termination of processes that we want to run even after the terminal is closed.
Terminal Multiplexer (
tmux) to start the
- Start the ssh terminal
tmux. It will open a window in the same terminal.
- Run the command to start long-running processes in the tmux session.
Now even if SSH terminal is closed/terminated suddenly tmux session will keep running the started lon-running processes on the instance/server.
If the connection terminated then how to reconnect it to see the processes running in the tmux session in the background:
- reconnect or open new ssh terminal. To see this process(which is kept running in the background) type:
Want to terminate tmux session:
- Stop the process. Then type
(Note that: If we use
tmux detach command: it will exit from tmux session window/terminal without terminating/stopping the tmux sessions)
For more details please refer following article:
In 2021 we can also suggest for linux distributions with systemd the use of
systemd-run as a way to detach a process and keep it running after closing ssh login.
One can launch a
command , give it some name like 'background_cmd_service' and turn it into a systemd service. Later inspect status:
systemd-run --unit=background_cmd_service --remain-after-exit command # later on journalctl -b -u background_cmd_service.service systemctl status background_cmd_service.service
When launching the service as regular user, one might need to enable lingering (cf.
enable-linger loginctl option). This is true even with nohup since systemd is cleaning up all services running within the session once the user logs out.
For scripts, or other executables, you can provide the path to the executable like this:
systemd-run --unit=background_cmd_service --remain-after-exit --working-directory=/full/path/ command
You can also stop your service or clear a failed service using the standard
systemctl options like:
systemctl stop background_cmd_service.service
systemctl reset-failed background_cmd_service.service
Many of the pre-2016, but highly voted answers on stack exchange suggesting
command &, and
(command &) don't work since 2016, due to a systemd default change in how it treats user processes after user logout - Details
I use NX NoMachine, which is free for me because it's only me. Essentially, it runs an X session on the server which you can connect to and disconnect from over and over. The X session keeps running when you're not connected. Connections can be made from anywhere. You can choose between floating windows or a single window containing a whole desktop (eg a complete Gnome desktop). The client (which you would run on your laptop) can be run on Linux, MacOS, Solaris or Microsoft Windows. In the latter case if you choose floating windows they appear individually on the Windows Taskbar.
I use my Windows XP laptop (which I need for certain Windows-specific hardware I have) as a front end for my two Linux servers using NX Nomachine. I can even print to the printer attached to my Windows laptop from Linux.
If you want to run the process in a non-interactive ssh session (so
screen are out) you can do:
ssh SERVER 'nohup bash -c "echo start; sleep 10; echo finish"'
The process will continue if the ssh session is interrupted but you cannot reconnect to see its output.
ssh SERVER 'nohup bash -c "(echo start; sleep 10; echo finish)|tee -p /tmp/log"'
This will show you the output during the ssh session and if it disconnects
tee -p will continue to write to the
Note: since the session is not interactive you can't use
& to put the job into the background.