8

Under Linux, I have a process that is blocked in uninterruptible sleep (state D). How can I investigate what's causing this?

I am running an “ordinary” kernel (a Debian build), without any special debugging features.

There is no relevant log entry — in fact nothing got logged between the time the process started and the time I noticed it.

strace can't even attach to the process since it's in uninterruptible sleep. And even if I knew what system call was called, that wouldn't necessarily help me. I need to know what's going on inside the kernel.

Specifically, the sync command goes into uninterruptible sleep :( So I must have an I/O problem somewhere but all my filesystems appear to work normally. There may well be an old log entry about an I/O error but I can't find it (this machine hasn't rebooted in a long time, that's a lot of log entries). Can I at least know which subsystem is blocking sync? For example, get a kernel backtrace for the kernel thread corresponding to a particular PID/TID?

(I'm sure that rebooting would either fix this or reveal the error but I'm asking how to investigate this, not how to blindly press a button.)

7
  • Are you using NFS? You may be able to find some information with lsof.
    – BillThor
    Aug 16, 2016 at 2:28
  • Do you have NFS / CIFS / iSCSI mounts? Do you have USB connected removable devices? To help track the bad mount, typically df will end up hanging on the bad mount in this type of scenario but not always if the problem is with writing and the read is properly cached. Aug 16, 2016 at 2:29
  • @StephenHarris df does not hang, even after flushing the disk cache. I happen not to have any network mounts — those would be my first culprit — but even if I did, I'd want to know which one. Aug 16, 2016 at 2:32
  • Is this reproducible across system reboots? Sep 6, 2016 at 10:53
  • @countermode No. (Maybe it could be reproduced by mounting a particular filesystem, reading from a particular disk bloc, or some such, but I have no idea what since I don't know what caused the problem in the first place.) And since the power failed the other day I can't investigate this specific problem any more. But the general question would remain. Sep 6, 2016 at 10:59

1 Answer 1

4

It's a bit late, but it could be helpful for others.

What I did:

  1. cat /proc/PID/stack to get some direction. In my case it was connected with inode and filesystem:
[<ffffffff83bbd6f1>] wait_on_page_bit+0x81/0xa0            
[<ffffffff83bced9b>] truncate_inode_pages_range+0x42b/0x750
[<ffffffff83bcf12f>] truncate_inode_pages_final+0x4f/0x60  
[<ffffffff83c6b78c>] evict+0x16c/0x180                     
[<ffffffff83c6bafc>] iput+0xfc/0x190                       
[<ffffffff83c66498>] __dentry_kill+0x158/0x1d0             
[<ffffffff83c66b35>] dput+0xb5/0x1a0                       
[<ffffffff83c4f53d>] __fput+0x18d/0x230                    
[<ffffffff83c4f6ce>] ____fput+0xe/0x10                     
[<ffffffff83ac31cb>] task_work_run+0xbb/0xe0               
[<ffffffff83a2cc65>] do_notify_resume+0xa5/0xc0            
[<ffffffff8419322f>] int_signal+0x12/0x17                  
[<ffffffffffffffff>] 0xffffffffffffffff                    
  1. cat /proc/PID/syscall to get current system call:
3 0x6 0x1ae4bc6d 0x1 0x559320c 0x801df5 0x60161c4e 0x7ffccee38ae0 0x7fcf1a1547bd

3 stands for close syscall, 6 is file descriptor (first argument of syscall). It was trying to call close(6).

  1. lsof -p PID, but there wasn't my descriptor.
  2. If you are lucky and your file is open somewhere at the start of your application, you can start another instance of it and inspect this file by lsof. It was my case.

Goodluck

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .