I'm trying to come up with a script for managing jobs on a supercomputer. The details don't matter much, but a key point is that the script starts to tail -f
a file once it appears. Now this would run forever, but I want to cleanly stop it and exit the script once I detect that the job is finished.
Unfortunately I'm stuck. I tried multiple solutions, but none of them exits the script, it keeps running even after the job was detected to exit. The version below seemed to be the most logical one, but this one too keeps running forever.
How should I tackle this issue? I'm OK with bash, but not really advanced.
#!/bin/bash
# get the path to the job script, print help if not passed
jobscr="$1"
[[ -z "$jobscr" ]] && echo "Usage: submit-and-follow [script to submit]" && exit -2
# submit job via SLURM (the job secluder), and get the
# job ID (4-5-digit number) from it's output, exit if failed
jobmsg=$(sbatch "$jobscr")
ret=$?
echo "$jobmsg"
if [ ! $ret -eq 0 ]; then exit $ret; fi
jobid=$(echo "$jobmsg" | cut -d " " -f 4)
# get the stdout and stderr file the job is using, we will log them in another
# file while we `tail -f` them (this is neccessary due to a file corruption
# bug in the supercomputer, just assume it makes sense)
outf="$(scontrol show job $jobid | awk -F= '/StdOut=/{print $2}')"
errf="$(scontrol show job $jobid | awk -F= '/StdErr=/{print $2}')"
logf="${outf}.bkp"
# wait for job to start
echo "### Waiting for job $jobid to start..."
until [ -f "$outf" ]; do sleep 5; done
# ~~~~ HERE COMES THE PART IN QUESTION ~~~~ #
# Once it started, start printing the content of stdout and stderr
# and copy them into the log file
echo "### Job $jobid started, stdout and stderr:"
tail -f -n 100000 $outf $errf | tee $logf &
tail_pid=$! # catch the pid of the child process
# watch for job end (the job id disappears from the queue; consider this
# detection working), and kill the tail process
while : ; do
sleep 5
if [[ -z "$(squeue | grep $jobid)" ]]; then
echo "### Job $jobid finished!"
kill -2 $tail_pid
break
fi
done
I also tried another version where tail
was in the main process, and the while
loop was running in a child process instead, which killed the main process once the job ended, but it didn't work out. Either way, the script never terminates.
tail | tee
in Bash runs a subshell, and does not return the Pid of either the tail or the tee. SigInt (kill -2) will be ignored by that subshell.$outf
to appear - you can usetail
's-F
flag (equivalent to--retry -f
) instead of-f
, which will keep trying to open the file until it appears.