How do I wait for a process to end and then do a series of actions (e.g. kill a tmux session) without blocking my main terminal?

Question

I am trying to "re-implement" nohup inside of tmux (so dispatching a job without blocking me and then killing the tmux command acting as nohup). My attempt is here (and it has several issues as outlined the beginning of it won't work but it's a seperate question, this question is about the last part of the script). My main issue right now is that I want to kill the tmux session once my real python script running in the background is done. This is my full attempt:

# - get a job id for this tmux session
export SLURM_JOBID=$(python -c "import random;print(random.randint(0, 1_000_000))")
echo SLURM_JOBID = $SLURM_JOBID
export OUT_FILE=$PWD/main.sh.o$SLURM_JOBID
export ERR_FILE=$PWD/main.sh.e$SLURM_JOBID

# - CAREFUL, if a job is already running it could do damage to it, rm reauth process, qian doesn't do it so skip it
# top -u brando9
# pkill -9 reauth -u brando9

# - start tmux, 
https://unix.stackexchange.com/questions/724877/custom-kerberos-tmux-doesnt-let-me-name-my-sessions-help-forcing-it
tmux new -s $SLURM_JOBID
# /afs/cs/software/bin/krbtmux new -s $SLURM_JOBID
# cat /afs/cs/software/bin/krbtmux

# - reauth
# /afs/cs/software/bin/reauth
echo $SU_PASSWORD | /afs/cs/software/bin/reauth
# echo 'Secret' | /afs/cs/software/bin/reauth
# echo 'totally secret password' | kinit [email protected]
# to see reauth running
# top -u brando9 

# - expt python script
python expts.py &
python -u ~/diversity-for-predictive-success-of-meta-learning/div_src/diversity_src/experiment_mains/main_sl_with_ddp.py --manual_loads_name sl_hdb1_5cnn_adam_cl_filter_size --filter_size 4 > $OUT_FILE 2> $ERR_FILE &
export JOB_PID=$!
echo JOB_PID = $JOB_PID

# - Echo tmux id, should be jobid
tmux display-message -p '#S'
echo SLURM_JOBID = $SLURM_JOBID

# - detach current tmux session
tmux detach &

# - wait for pid from python to be done, if done kill this tmux sess
wait $JOB_PID
tmux ls
tmux kill-session -t $SLURM_JOBID
# check it was killed
tmux ls

# - Done
echo "Done with bash script (experiment or dispatched daemon experiments). "

But the above won't work because: once the python script is running and been dispatched with &, what I want is to kill the tmux session when that job is done. I tried that at the end but I suspect it won't work. I think it won't work because once it runs tmux detach & it will go out of the tmux session and run the wait ... command outside of tmux and block my main terminal. Instead what I want is to run that wait command inside of tmux (to not block me) and once the python script is done to kill the tmux session entirely. But that is just to add context to the final part of my script.

How do I make this work properly?

related: How does one authenticate with a command that requires your password in linux?

Answer 1

You can use the ; operators to chain commands, along with the & operator to send them to the background. Enclose the commands in parentheses to send the entire group into the background. Here is a demonstration:

tmux new -s test
(sleep 15 ; tmux kill-session -t test) &
tmux detach

while true ; do 
  if ! tmux ls 2> /dev/null | grep 'test:' ; then
    echo tmux test disconnected
    break
  fi
  sleep 1 # limit polling rate
done

Answer 2

You cannot use wait in this particular way, as wait requires parent-child relationship between processes. Using ; will not work either as that is simply a command separator, i.e. the command will have to complete until your prompt is returned to you.

Instead, I believe what you are looking for is simply something akin to the following example:

### Random SLURM_JOBID value
export SLURM_JOBID=$(python3 -c "import random;print(random.randint(0, 1_000_000))")

### Init tmux session
tmux new -s "$SLURM_JOBID"

### --- tmux session ---
### Test command: runs in the background of a detached tmux session 
### and once finished will self-destruct that same detached tmux session

(for  i in $(seq 1 30); do sleep 1 ; done && tmux kill-session -t "$SLURM_JOBID" ) & tmux detach

Once you execute the above command, you will be immediately returned to your tmux-originating shell.

Executing - inside of the 30-second runtime of the loop - the following command will confirm that your tmux is indeed still present in the background ( you will even be able to still reattach to it with tmux attach)

> ps aux | grep tmux
user       12579  0.0  0.0   5880  3928 ?        Ss   10:26   0:00 tmux
user       12613  0.0  0.0   3876  1836 pts/0    S+   10:27   0:00 grep --color=auto tmux

> tmux list-sessions
683635: 1 windows (created Wed Nov 30 10:26:02 2022)

Executing the same command after the 30 seconds are up will confirm that its parent tmux session no longer exists.

> ps aux | grep tmux
user       12706  0.0  0.0   3876  1836 pts/0    S+   10:30   0:00 grep --color=auto tmux

> tmux list-sessions
no server running on /tmp/tmux-0/default

---

---

Command syntax explanation

(for  i in $(seq 1 30); do sleep 1 ; done && tmux kill-session ) & tmux detach

There are several unique aspects of the above command and I will try to go over them one by one:

• Part 1: [CMD1] && [CMD2]

in the context of the above command, [CMD1] and [CMD2] correspond to the following:

[CMD1] = for i in $(seq 1 30); do sleep 1 ; done

[CMD2] = tmux kill-session -t "$SLURM_JOBID"

Using the double ampersand between two different commands, i.e. [CMD1] && [CMD2] forces the shell to wait and execute [CMD2] if AND only if [CMD1] completes successfully.

In other words, in the context of our command the shell is being told to start the loop, run 30 iterations of sleep 1 and once the loop is finished, ONLY THEN execute tmux kill-session.

You can test this behavior for yourself, by executing - inside of a tmux session - the following command combo:

for  i in $(seq 1 10); do sleep 1 ; done && tmux kill-session 

The loop will start, and run in the foreground, placing a hold on your prompt.

If you let it run for 10 seconds and finish, it will then execute tmux kill-session found in the portion after && and destroy its own tmux session returning you back to the previous shell.

On the other hand, if you send CTRL+z and kill the loop mid-execution, your tmux session will survive to live another day because the shell will register our SIGINT as an unsuccessful [CMD1] and not allow the [CMD2] on the right side of the && to execute.

---

• Part 2: ( [CMD1] && [CMD2] )

The parenthesis around multiple commands allows for their grouping and isolation, resulting in something similar to the behavior of mathematical order of operations.

---

• Part 3: ( [CMD1] && [CMD2] ) &

This addition of the ampersand & after the parenthesis allows us to execute the entirety of what's enclosed inside the parenthesis in the background.

Without the use of (...) execution of multiple commands in the background becomes a lot more tedious and complicated, as one cannot for example simply use [CMD1] & && [CMD2] or [CMD1] && & [CMD2]

On the other hand, using (...) one can execute even something as complex as the following:

((for  i in $(seq 1 5); do sleep 1 ; done && echo "first loop finished" )  && ( for i in $(seq 1 5); do sleep 1; done && echo "second loop finished" ) && sleep 5 && tmux kill-session ) &

which will execute the first loop, once done its status will be reported then the second loop will start, and once that is done its status will be reported as well , and then there will be a 5 second sleep delay, and then the session will self destruct.

• Part 4: ([CMD1] && [CMD2]) & tmux detach

One last addition is that of tmux detach which allows us to return to the shell that originated our tmux session. One thing worth mentioning that one could also get the same behavior by running on two separte lines the following two commands:

(for  i in $(seq 1 30); do sleep 1 ; done && tmux kill-session -t "$SLURM_JOBID" ) &
tmux detach

one after another, as the single & belongs to the left side of our one-liner.