2

I'm running a bash script which contains a for loop which will run thousands of times. Somewhere in this for loop the code often gets stuck (lowest number of iterations I have seen it stop on is 32, most it has stopped on is in the high 4000s - it is supposed to run 5000 times, sometimes it completes the task). I would like to debug this problem. Obviously, I don't want to manually enters hundreds of thousands of lines of code into the command window and wait for the one where it will get stuck.

Is there a simple way to find out what the last line it executed or tried to execute was? (Bearing in mind that when it goes wrong the terminal appears to be frozen - sometimes the entire system stops working and requires an off-at-the-wall reset to get it started again.) I thought this might be a feature of echo. I found this question and answer but I am not sure what "when commands represent a specific line in my script from a parser standpoint" means and therefore if it is applicable. I'm also pretty confused by what this code actually does and would appreciate for dummies explanation if this code will help me.

Is this the answer I am looking for? Is there another solution to this problem?

  • 1
    This is what debuggers are useful for. Searching for 'bash debuggers' should net you a lot of results. A bit of warning, debuggers can be complicated, but there are lots of resources for learning. In this case, I do not think using the echo command will benefit you in any way inside the for-loop. – cremefraiche Dec 11 '14 at 14:43
  • Good to know, I'll stop looking into echo-related material. – Reluctant_Linux_User Dec 11 '14 at 15:04
  • Not that there is anything wrong with echo, you will probably use it often. I would just recommend caution when running additional commands inside of a loop with that many iterations, especially if it is already crashing sporadically. A small script can very quickly become resource intensive. Just ask the fork bomb. – cremefraiche Dec 11 '14 at 15:11
  • Does the script process 1 iteration successfully? – eyoung100 Dec 11 '14 at 15:28
  • @eyoung100 Yes, it runs many times, sometimes it even completes. – Reluctant_Linux_User Dec 11 '14 at 15:36
3

First you should try to determine in which iteration the problem occurs. If you are lucky then it is always the same. If you do not do that anyway then you should count the iterations and print them or write them to a file:

exec 3>/my/logfile
iter=0
while whatevercondition; do
  ((iter++))
  echo "$iter" >&3
done

If the problem occurs always in the same iteration then you should activate debug output then:

CRASH_ITER=12345
iter=0
while whatevercondition; do
  ((iter++))
  if [ "$iter" -eq "$CRASH_ITER" ]; then
    set -vx
  fi
done

non-reproducible errors

If it is not possible to narrow the error to a certain iteration then you can have bash write the complete debug output to a file:

bash -vx ./script.sh 2>debug.txt
  • Unfortunately the iteration the error occurs on is not always the same It is not always the same seen numbers anywhere between 32 and 4885 out of 5000 iterations. Sometimes the code even completes! – Reluctant_Linux_User Dec 11 '14 at 15:05
  • @Reluctant_Linux_User I extended my answer accordingly. – Hauke Laging Dec 11 '14 at 15:09
  • I take it that I am using that as a command to start the bash script - yes? – Reluctant_Linux_User Dec 11 '14 at 15:12
  • @Reluctant_Linux_User Yes. It has the same effect like set -vx within a script but is usually easier to change. – Hauke Laging Dec 11 '14 at 15:14
  • OK, well the code just crashed in fact it's crashed the entire system (writing this on a stable system) so I guess I can give that a go. – Reluctant_Linux_User Dec 11 '14 at 15:16
1

If you add this to the top of your bash script:


   function DEBUG_TRAP() { 
      typeset -p BASH_SOURCE; typeset -p BASH_LINENO; typeset -p FUNCNAME
   }
   trap DEBUG_TRAP INT

you may be able to get information about where the program is by sending the process the an INT signal:

kill -INT process-number

You can get the process number inside the bash script like this:

   echo $$

If you aren't getting any output after doing this then either your script has redirected its output, or the script is hung inside a process. lsof will tell you information about output redirection while ps will give information on the process status.

0

If your loop looked like:

i=0
while [ "$((i+=1))" -le 5000 ] &&
      set >iter.log
do    monte_carlo
done

You'd get a file overwritten for each iteration which would list the current values of all shell variables each time. So if it breaks on run 488 you'd have have an iter.log file which recorded $i's value as 488 and all of the other shell variables at the beginning of the iteration.

To track the failing line number you can go a little further:

i=0 PS4='$LINENO : '; set -x
while [ ... ] &&
      set >iter.log
do    ...
done 2>>iter.log

That would still overwrite the file for every iteration, but it would also append debug output for every line executed and include its line number in the file.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.