I'm running a bash script which contains a for loop which will run thousands of times. Somewhere in this for loop the code often gets stuck (lowest number of iterations I have seen it stop on is 32, most it has stopped on is in the high 4000s - it is supposed to run 5000 times, sometimes it completes the task). I would like to debug this problem. Obviously, I don't want to manually enters hundreds of thousands of lines of code into the command window and wait for the one where it will get stuck.

Is there a simple way to find out what the last line it executed or tried to execute was? (Bearing in mind that when it goes wrong the terminal appears to be frozen - sometimes the entire system stops working and requires an off-at-the-wall reset to get it started again.) I thought this might be a feature of echo. I found this question and answer but I am not sure what "when commands represent a specific line in my script from a parser standpoint" means and therefore if it is applicable. I'm also pretty confused by what this code actually does and would appreciate for dummies explanation if this code will help me.

Is this the answer I am looking for? Is there another solution to this problem?

  • 1
    This is what debuggers are useful for. Searching for 'bash debuggers' should net you a lot of results. A bit of warning, debuggers can be complicated, but there are lots of resources for learning. In this case, I do not think using the echo command will benefit you in any way inside the for-loop. Commented Dec 11, 2014 at 14:43
  • Good to know, I'll stop looking into echo-related material. Commented Dec 11, 2014 at 15:04
  • Not that there is anything wrong with echo, you will probably use it often. I would just recommend caution when running additional commands inside of a loop with that many iterations, especially if it is already crashing sporadically. A small script can very quickly become resource intensive. Just ask the fork bomb. Commented Dec 11, 2014 at 15:11
  • Does the script process 1 iteration successfully?
    – eyoung100
    Commented Dec 11, 2014 at 15:28
  • @eyoung100 Yes, it runs many times, sometimes it even completes. Commented Dec 11, 2014 at 15:36

3 Answers 3


First you should try to determine in which iteration the problem occurs. If you are lucky then it is always the same. If you do not do that anyway then you should count the iterations and print them or write them to a file:

exec 3>/my/logfile
while whatevercondition; do
  echo "$iter" >&3

If the problem occurs always in the same iteration then you should activate debug output then:

while whatevercondition; do
  if [ "$iter" -eq "$CRASH_ITER" ]; then
    set -vx

non-reproducible errors

If it is not possible to narrow the error to a certain iteration then you can have bash write the complete debug output to a file:

bash -vx ./script.sh 2>debug.txt
  • Unfortunately the iteration the error occurs on is not always the same It is not always the same seen numbers anywhere between 32 and 4885 out of 5000 iterations. Sometimes the code even completes! Commented Dec 11, 2014 at 15:05
  • @Reluctant_Linux_User I extended my answer accordingly. Commented Dec 11, 2014 at 15:09
  • I take it that I am using that as a command to start the bash script - yes? Commented Dec 11, 2014 at 15:12
  • @Reluctant_Linux_User Yes. It has the same effect like set -vx within a script but is usually easier to change. Commented Dec 11, 2014 at 15:14
  • OK, well the code just crashed in fact it's crashed the entire system (writing this on a stable system) so I guess I can give that a go. Commented Dec 11, 2014 at 15:16

If you add this to the top of your bash script:

   function DEBUG_TRAP() { 
      typeset -p BASH_SOURCE; typeset -p BASH_LINENO; typeset -p FUNCNAME

you may be able to get information about where the program is by sending the process the an INT signal:

kill -INT process-number

You can get the process number inside the bash script like this:

   echo $$

If you aren't getting any output after doing this then either your script has redirected its output, or the script is hung inside a process. lsof will tell you information about output redirection while ps will give information on the process status.


If your loop looked like:

while [ "$((i+=1))" -le 5000 ] &&
      set >iter.log
do    monte_carlo

You'd get a file overwritten for each iteration which would list the current values of all shell variables each time. So if it breaks on run 488 you'd have have an iter.log file which recorded $i's value as 488 and all of the other shell variables at the beginning of the iteration.

To track the failing line number you can go a little further:

i=0 PS4='$LINENO : '; set -x
while [ ... ] &&
      set >iter.log
do    ...
done 2>>iter.log

That would still overwrite the file for every iteration, but it would also append debug output for every line executed and include its line number in the file.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .