0

Recently I see these messeges in my /var/log/messeges and dmesg | less:

 kernel: INFO: task flush-8:32:1065 blocked for more than 120 seconds.
 kernel:      Not tainted 2.6.32-696.el6.x86_64 #1
 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this 
 message.
 kernel: flush-8:32    D 0000000000000004     0  1065      2 0x00000000
 kernel: ffff88063389b840 0000000000000046 0000000000000000 0005120000000001
 kernel: ffff880636304ce0 ffff8806363098c0 00022e760ddb7d3a ffff88062db31360
 kernel: 0000000000011200 000000012494c97a ffff88062e0fbad8 ffff88063389bfd8
 kernel: Call Trace:
 kernel: [<ffffffff811d11e0>] ? sync_buffer+0x0/0x50
 kernel: [<ffffffff8154ae83>] io_schedule+0x73/0xc0
 kernel: [<ffffffff811d1220>] sync_buffer+0x40/0x50
 kernel: [<ffffffff8154b73a>] __wait_on_bit_lock+0x5a/0xc0
 kernel: [<ffffffff811d11e0>] ? sync_buffer+0x0/0x50
 kernel: [<ffffffff8154b818>] out_of_line_wait_on_bit_lock+0x78/0x90
 kernel: [<ffffffff810a68c0>] ? wake_bit_function+0x0/0x50
 kernel: [<ffffffff811d1540>] ? end_buffer_async_write+0x0/0x190
 kernel: [<ffffffff811d13c6>] __lock_buffer+0x36/0x40
 kernel: [<ffffffff811d26c5>] __block_write_full_page+0x305/0x330
 kernel: [<ffffffff811d1540>] ? end_buffer_async_write+0x0/0x190
 kernel: [<ffffffff811d27d0>] block_write_full_page_endio+0xe0/0x120
 kernel: [<ffffffffa0211d00>] ? buffer_unmapped+0x0/0x20 [ext3]
 kernel: [<ffffffff811d2825>] block_write_full_page+0x15/0x20
 kernel: [<ffffffffa021288d>] ext3_ordered_writepage+0x1ed/0x240 [ext3]
 kernel: [<ffffffff81142367>] __writepage+0x17/0x40
 kernel: [<ffffffff8114362d>] write_cache_pages+0x1fd/0x4c0
 kernel: [<ffffffff81009913>] ? __switch_to+0x2f3/0x340
 kernel: [<ffffffff81142350>] ? __writepage+0x0/0x40
 kernel: [<ffffffff81299899>] ? cpumask_next_and+0x29/0x50
 kernel: [<ffffffff81064054>] ? find_busiest_group+0x254/0xa50
 kernel: [<ffffffff81143914>] generic_writepages+0x24/0x30
 kernel: [<ffffffff81143955>] do_writepages+0x35/0x40
 kernel: [<ffffffff811c6f1d>] writeback_single_inode+0xdd/0x290
 kernel: [<ffffffff811c731d>] writeback_sb_inodes+0xbd/0x170
 kernel: [<ffffffff811c747b>] writeback_inodes_wb+0xab/0x1b0
 kernel: [<ffffffff811c7873>] wb_writeback+0x2f3/0x410
 kernel: [<ffffffff8154a68e>] ? schedule+0x3ee/0xb70
 kernel: [<ffffffff811c7b3d>] wb_do_writeback+0x1ad/0x250
 kernel: [<ffffffff8108f000>] ? process_timeout+0x0/0x10
 kernel: [<ffffffff811c7c86>] bdi_writeback_thread+0xa6/0x220
 kernel: [<ffffffff811c7be0>] ? bdi_writeback_thread+0x0/0x220
 kernel: [<ffffffff810a63ae>] kthread+0x9e/0xc0
 kernel: [<ffffffff8100c28a>] child_rip+0xa/0x20
 kernel: [<ffffffff810a6310>] ? kthread+0x0/0xc0
 kernel: [<ffffffff8100c280>] ? child_rip+0x0/0x20  

Today one of VM machines rebooted. When I checked messeges, I saw above messeges and then some database errors. That network had a problem and this machine lost connection with its primary and just rebooted. My question is: what are these kernel messeges and where should I search for problem? Do they have something to do with network? Does it only happens on VM machines?

2

That warning is due to resource starvation.

If in a VM, it might be a clue the host is oversubscribed.

Mind you it is a warning (hence the INFO keyword), it can be ignored. However I advise dealing with the underlying issues that are causing it, also because they are bound to worsen over time.

I would go for lack or resources/overuse of I/O storage.

You can also try to run:

sudo sysctl -w vm.dirty_ratio=10
sudo sysctl -w vm.dirty_background_ratio=5

If that works out for you, put it permanently, for applying it at boot time, in /etc/sysctl.conf:

vm.dirty_background_ratio = 5
vm.dirty_ratio = 10

See Linux Kernel panic issue: How to fix hung_task_timeout_secs and blocked for more than 120 seconds problem

By default Linux uses up to 40% of the available memory for file system caching. After this mark has been reached the file system flushes all outstanding data to disk causing all following IOs going synchronous. For flushing out this data to disk this there is a time limit of 120 seconds by default. In the case here the IO subsystem is not fast enough to flush the data withing 120 seconds. As IO subsystem responds slowly and more requests are served, System Memory gets filled up resulting in the above error, thus serving HTTP requests.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.