nohup nice job aborted when disconnected

Question

I've learned that the nohup command allows the command to keep running if you lose your connection or need to logout. I'm running nohup nice my command & in remoter through ssh, and after my laptop lost its network connection, the terminal says "broken pipe" and exits automatically. When I got reconnected, the job is missing. Why is it like this? Is it possible to run it from where it's left off?

Answer 1

There's a few things to understand here:

1. nohup doesn't necessarily keep the program from losing the connection. What it does is mask the HUP signal, which is what the OS will send it (actually, the controlling shell) when you log off. If your program still has file descriptors connected to the terminal, it will get stopped and possibly killed. To prevent that from happening, nohup normally redirects stdout and stderr to nohup.out and closes stdin. But if your program may try to open file descriptors to the terminal anyway.

2. The shell doesn't keep track of "jobs" across invocations / sessions. There is no way to re-attach a job to a shell that has been exited. There are several work-arounds:

   1. Use a terminal emulator such as screen or tmux which can be re-attached to a different terminal. Here, you don't need to run your job in the background at all; just run it in the foreground and detach (Ctrl-A d with screen).

   2. Keep track of the PID of the job that was started. You can get this information with the jobs -l command after it has started. When you want the new shell to wait for the job, and you know the job's pid, you can issue the command:

      wait <pid>
      

But output will still be sent to nohup.out, and your only clue that it has finished is that you see a command prompt again.