3

We are using beagle bone black based custom board, with kernel version 3.12.
We are facing system hang during one of the init script,(Which brings up WiFi)
this hang happens after random number of power cycle,

Nothing works during this hang, it looks like system is frozen, It doesn't even respond to sysrq keys

I assume this to be in the ISR code because of which none of the thing works.

Unluckily When we enable 'Detect hung task(DETECT_HUNG_TASK)' we don't see the issue. :(

Only thing works is if watchdog is enabled after watchdog timer expires it reboots the system and system recovers.

However we want to find out where the issue is.

Any suggestion?

I thought of using softdog and repair script pair to print some messages but I assume external interrupt will have higher priority and when it executes and hangs in there, softdog timer will also not get a chance to execute right ?

Randomness of the bug makes it much more difficult to debug :(

Any help appreciated.

17
  • Did you look carefully at the logs when you rebooted after it hung? Jun 6 '16 at 6:23
  • Yes, We don't see anything there :( We doubt WiFI over SDIO driver, but we are not sure about it. Because when we don't load WiFi driver module we don't see it. However sometimes we don't see it even when it is enabled.
    – ART
    Jun 6 '16 at 6:32
  • @JuliePelletier Logs usually cannot be written after a hard hang, similar to kernel panic
    – user140866
    Jun 6 '16 at 6:46
  • 1
    Did you try to redirect console to serial, and boot the kernel with loglevel=7 without quiet (if any)? Are there some obscure messages from kernel were coming?
    – user140866
    Jun 6 '16 at 6:47
  • 2
    DETECT_HUNG_TASK is usually for userspace tasks that hang inside system call. If hang comes from kernel code (for example, driver), it is useless.
    – user140866
    Jun 6 '16 at 6:52
0

Well, We did code reading as it was being suggested in the comments and found the section of the patch where system may go into infinite loop(in irq) and won't come out of it.

However when we put printk in that irq function issue was not getting reproduced. (timing issue you know!)

So finally my colleague tried old school method of toggling GPIO and it helped. That was also difficult as more than two entries of GPIO toggle would prevent reproducing issue.

inside function he used GPIO toggle as follows,

func()
{  
   //set gpio high
     some doubtfull code..
     ....
   //set gpio low
}  

That's how he tracked the problematic code and its solution is available in linux-4.1 he fixed it and he is testing it.

@ShankarSM:If you are reading this, all credit goes to you for tracking down it :-)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.