2

I'm running Ubuntu 16.04.5 and the ext4 filesystem, and I have a file which causes processes to hang when they read it.

For example:

$ cat /path/to/file.txt

and then from a separate terminal:

$ sudo cat /proc/24147/stack
[<ffffffff811937dd>] wait_on_page_bit_killable+0xcd/0xf0
[<ffffffff81193c86>] generic_file_read_iter+0x486/0x6b0
[<ffffffff8121395e>] new_sync_read+0x9e/0xe0
[<ffffffff812139c9>] __vfs_read+0x29/0x40
[<ffffffff81213f96>] vfs_read+0x86/0x130
[<ffffffff81214ce5>] SyS_read+0x55/0xc0
[<ffffffff81829d4e>] entry_SYSCALL_64_fastpath+0x22/0xc1
[<ffffffffffffffff>] 0xffffffffffffffff

I have many processes hung reading this file. Here's another example with a different call stack:

$ less /path/to/file.txt

and then from a separate terminal:

$ sudo cat /proc/23006/stack
[<ffffffff8140f954>] call_rwsem_down_read_failed+0x14/0x30
[<ffffffff812a1e83>] ext4_map_blocks+0x443/0x5a0
[<ffffffff812edb98>] ext4_mpage_readpages+0x368/0x920
[<ffffffff8129f626>] ext4_readpages+0x36/0x40
[<ffffffff811a0b89>] __do_page_cache_readahead+0x199/0x240
[<ffffffff811a0d6d>] ondemand_readahead+0x13d/0x250
[<ffffffff811a101e>] page_cache_sync_readahead+0x2e/0x50
[<ffffffff81193d4a>] generic_file_read_iter+0x54a/0x6b0
[<ffffffff8121395e>] new_sync_read+0x9e/0xe0
[<ffffffff812139c9>] __vfs_read+0x29/0x40
[<ffffffff81213f96>] vfs_read+0x86/0x130
[<ffffffff81214ce5>] SyS_read+0x55/0xc0
[<ffffffff81829d4e>] entry_SYSCALL_64_fastpath+0x22/0xc1
[<ffffffffffffffff>] 0xffffffffffffffff

I need to narrow down the actual problem a bit. The context is somewhat involved (I'm running Apache and using mod_wsgi, and then Python code is writing to the file which is a log file, this volume is a RAID array on top of the instance store on an AWS instance etc, etc).

Is it possible to get a linux box into a state where cat'ing a file causes the terminal to hang as I'm showing above. From there I may determine what additional context (if any) would be useful here.

I should mention that this happens on a certain production machine (which sees heavy use) every month or so. I can restart the machine to recover, but I'm interested in understanding this state which it's in. Ideally I'd like to prevent this from happening in the first place.

  • I believe this is the same: unix.stackexchange.com/questions/350761/… – V13 Jan 21 at 22:27
  • @V13 I added another example from the same machine with a different call stack. I'm not sure it's the same problem as the other question you referenced, but I guess that's the thing - I have no idea what it could be and was hoping for some input to narrow it down a bit. – nonagon Jan 22 at 2:04
  • Can you SIGKILL that process? if not, then they're the same question, your filesystem or disk is broken, try to do offline fsck. – 炸鱼薯条德里克 Jan 22 at 2:08
  • I can kill the cat process but not the less one – nonagon Jan 22 at 16:20
  • Did you try fsck? It looks like a corrupted ext4 filesystem that triggers a "buggy" codepath that's not normally exercised – V13 Jan 24 at 0:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.