Is there a simple way to find out the last line executed in a bash script?

Question

I'm running a bash script which contains a for loop which will run thousands of times. Somewhere in this for loop the code often gets stuck (lowest number of iterations I have seen it stop on is 32, most it has stopped on is in the high 4000s - it is supposed to run 5000 times, sometimes it completes the task). I would like to debug this problem. Obviously, I don't want to manually enters hundreds of thousands of lines of code into the command window and wait for the one where it will get stuck.

Is there a simple way to find out what the last line it executed or tried to execute was? (Bearing in mind that when it goes wrong the terminal appears to be frozen - sometimes the entire system stops working and requires an off-at-the-wall reset to get it started again.) I thought this might be a feature of echo. I found this question and answer but I am not sure what "when commands represent a specific line in my script from a parser standpoint" means and therefore if it is applicable. I'm also pretty confused by what this code actually does and would appreciate for dummies explanation if this code will help me.

Is this the answer I am looking for? Is there another solution to this problem?

Answer 1

First you should try to determine in which iteration the problem occurs. If you are lucky then it is always the same. If you do not do that anyway then you should count the iterations and print them or write them to a file:

exec 3>/my/logfile
iter=0
while whatevercondition; do
  ((iter++))
  echo "$iter" >&3
done

If the problem occurs always in the same iteration then you should activate debug output then:

CRASH_ITER=12345
iter=0
while whatevercondition; do
  ((iter++))
  if [ "$iter" -eq "$CRASH_ITER" ]; then
    set -vx
  fi
done

non-reproducible errors

If it is not possible to narrow the error to a certain iteration then you can have bash write the complete debug output to a file:

bash -vx ./script.sh 2>debug.txt

Answer 2

If you add this to the top of your bash script:

   function DEBUG_TRAP() { 
      typeset -p BASH_SOURCE; typeset -p BASH_LINENO; typeset -p FUNCNAME
   }
   trap DEBUG_TRAP INT

you may be able to get information about where the program is by sending the process the an INT signal:

kill -INT process-number

You can get the process number inside the bash script like this:

   echo $$

If you aren't getting any output after doing this then either your script has redirected its output, or the script is hung inside a process. lsof will tell you information about output redirection while ps will give information on the process status.

Answer 3

If your loop looked like:

i=0
while [ "$((i+=1))" -le 5000 ] &&
      set >iter.log
do    monte_carlo
done

You'd get a file overwritten for each iteration which would list the current values of all shell variables each time. So if it breaks on run 488 you'd have have an iter.log file which recorded $i's value as 488 and all of the other shell variables at the beginning of the iteration.

To track the failing line number you can go a little further:

i=0 PS4='$LINENO : '; set -x
while [ ... ] &&
      set >iter.log
do    ...
done 2>>iter.log

That would still overwrite the file for every iteration, but it would also append debug output for every line executed and include its line number in the file.