1

I have a 2 large mdadm arrays, and I combine them in a volume group. I was recently adding new drives to the second array and I suffered a power outage. Typically, I don't have major problems getting arrays to restart or resume a reshape after an outage, but this time I am having a lot of trouble.

System details: CentOS 6.8 x64

Linux myserver 2.6.32-642.4.2.el6.x86_64 #1 SMP Tue Aug 23 19:58:13 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

When I first boot up, the array assembles but doesn't start because it is degraded. I can't get it started with:

mdadm -R /dev/md126

I then stop it:

mdadm -S /dev/md126

If I assemble it normally, it won't start:

mdadm --assemble --scan
mdadm: device 26 in /dev/md/128 has wrong state in superblock, but /dev/sdn seems ok
mdadm: /dev/md/128 assembled from 12 drives and 2 spares - not enough to start the array while not clean - consider --force.
mdadm: No arrays found in config file or automatically

so I assemble with:

mdadm --assemble --scan --force
mdadm: clearing FAULTY flag for device 1 in /dev/md/128 for /dev/sdn
mdadm: Marking array /dev/md/128 as 'clean'
mdadm: /dev/md/128 has been started with 12 drives (out of 13) and 2 spares.

At this point, my session is hung. But if I ssh back into the device I can run commands while the other session continues to hang.

cat /proc/mdstat

md128 : active raid6 sdf[0] sdl[9](S) sdn[14](S) sdo[6] sdm[7] sdc[10] sdb[11] sdd[12] sde[13] sdk[5] sdj[4] sdi[3] sdh[2] sdg[1]
      23441323008 blocks super 1.2 level 6, 512k chunk, algorithm 2 [13/12] [UUUUUUUUUUUU_]
      [>....................]  reshape =  0.0% (128496/3906887168) finish=133658963.6min speed=0K/sec
      bitmap: 1/30 pages [4KB], 65536KB chunk

md127 : active (auto-read-only) raid6 sdaa[2] sdab[13] sdac[0] sdad[15] sdy[4] sdx[5] sdz[14] sdw[6] sdr[9] sds[8] sdq[10] sdp[16] sdu[11] sdv[7] sdt[12]
      50789533184 blocks super 1.2 level 6, 512k chunk, algorithm 2 [15/15] [UUUUUUUUUUUUUUU]

unused devices: <none>

The reshape never makes it past that point. It's always stuck there.

In the dmesg I then get the following error

created bitmap (30 pages) for device md128
md128: bitmap initialized from disk: read 2 pages, set 1 of 59615 bits
md128: detected capacity change from 0 to 24003914760192
md: reshape of RAID array md128
md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for reshape.
 md128:
md: using 128k window, over a total of 3906887168k.
 unknown partition table
INFO: task mdadm:2357 blocked for more than 120 seconds.
      Not tainted 2.6.32-642.4.2.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
mdadm         D 0000000000000002     0  2357   2179 0x00000004
 ffff880821773688 0000000000000082 0000000000000000 0000000000000086
 ffff880821773618 ffffffff810633a3 0000013e9de36948 ffff88081f96d440
 ffff88081dc65ed0 000000010010516a ffff88081d713068 ffff880821773fd8
Call Trace:
 [<ffffffff810633a3>] ? __wake_up+0x53/0x70
 [<ffffffffa0240805>] get_active_stripe+0x2d5/0x880 [raid456]
 [<ffffffff81130d00>] ? mempool_alloc+0x20/0x140
 [<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff810a6bce>] ? prepare_to_wait+0x4e/0x80
 [<ffffffffa02455ef>] make_request+0x19f/0xcb0 [raid456]
 [<ffffffff812ab7de>] ? __sg_alloc_table+0x7e/0x130
 [<ffffffff81056cc5>] ? gup_pte_range+0xe5/0x130
 [<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff8143066f>] md_make_request+0xdf/0x240
 [<ffffffff8127ddb0>] generic_make_request+0x240/0x5a0
 [<ffffffff811d941c>] ? do_direct_IO+0x57c/0xfa0
 [<ffffffff811d60cb>] ? bio_alloc_bioset+0x5b/0xf0
 [<ffffffff8127e180>] submit_bio+0x70/0x120
 [<ffffffff811daabd>] __blockdev_direct_IO_newtrunc+0xc7d/0x1270
 [<ffffffff811d6320>] ? blkdev_get_block+0x0/0x20
 [<ffffffff811db127>] __blockdev_direct_IO+0x77/0xe0
 [<ffffffff811d6320>] ? blkdev_get_block+0x0/0x20
 [<ffffffff811d73a7>] blkdev_direct_IO+0x57/0x60
 [<ffffffff811d6320>] ? blkdev_get_block+0x0/0x20
 [<ffffffff8113028b>] generic_file_aio_read+0x6bb/0x700
 [<ffffffff8143877f>] ? md_ioctl+0x31f/0x1ac0
 [<ffffffff811d7f00>] ? blkdev_open+0x0/0xc0
 [<ffffffff81196d47>] ? __dentry_open+0x257/0x380
 [<ffffffff811d6891>] blkdev_aio_read+0x51/0x80
 [<ffffffff81199a6a>] do_sync_read+0xfa/0x140
 [<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff811d66bc>] ? block_ioctl+0x3c/0x40
 [<ffffffff811af522>] ? vfs_ioctl+0x22/0xa0
 [<ffffffff811af6c4>] ? do_vfs_ioctl+0x84/0x580
 [<ffffffff8123aa06>] ? security_file_permission+0x16/0x20
 [<ffffffff8119a365>] vfs_read+0xb5/0x1a0
 [<ffffffff8119b116>] ? fget_light_pos+0x16/0x50
 [<ffffffff8119a6b1>] sys_read+0x51/0xb0
 [<ffffffff8100b0d2>] system_call_fastpath+0x16/0x1b
INFO: task md128_reshape:2428 blocked for more than 120 seconds.
      Not tainted 2.6.32-642.4.2.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
md128_reshape D 0000000000000001     0  2428      2 0x00000000
 ffff88081fdcfa60 0000000000000046 0000000000000000 0000000000000086
 ffff88081fdcf9f0 ffffffff810633a3 0000013d983220e0 ffff88081f96d440
 ffff88081dc65ed0 0000000100103f9f ffff88081ea1e5f8 ffff88081fdcffd8
Call Trace:
 [<ffffffff810633a3>] ? __wake_up+0x53/0x70
 [<ffffffffa0240805>] get_active_stripe+0x2d5/0x880 [raid456]
 [<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffffa02447c0>] reshape_request+0x440/0xa20 [raid456]
 [<ffffffff810633a3>] ? __wake_up+0x53/0x70
 [<ffffffffa02450b2>] sync_request+0x312/0x3a0 [raid456]
 [<ffffffff81430ff7>] md_do_sync+0x6c7/0xd60
 [<ffffffff81431ae5>] md_thread+0x115/0x150
 [<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff814319d0>] ? md_thread+0x0/0x150
 [<ffffffff810a640e>] kthread+0x9e/0xc0
 [<ffffffff8100c28a>] child_rip+0xa/0x20
 [<ffffffff810a6370>] ? kthread+0x0/0xc0
 [<ffffffff8100c280>] ? child_rip+0x0/0x20
INFO: task blkid:2430 blocked for more than 120 seconds.
      Not tainted 2.6.32-642.4.2.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
blkid         D 0000000000000002     0  2430      1 0x00000000
 ffff8808211d36f8 0000000000000082 0000000000000000 0000000000000082
 ffff8808211d3688 ffffffff810633a3 0000013d9831af63 ffff88081f96d440
 ffff88081dc65ed0 0000000100104045 ffff88081e615ad8 ffff8808211d3fd8
Call Trace:
 [<ffffffff810633a3>] ? __wake_up+0x53/0x70
 [<ffffffffa0240805>] get_active_stripe+0x2d5/0x880 [raid456]
 [<ffffffff8142af00>] ? try_module_get+0x30/0xb0
 [<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff810a6bce>] ? prepare_to_wait+0x4e/0x80
 [<ffffffffa02455ef>] make_request+0x19f/0xcb0 [raid456]
 [<ffffffff81340b4b>] ? mix_pool_bytes_extract+0x16b/0x180
 [<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff81151319>] ? zone_statistics+0x99/0xc0
 [<ffffffff8143066f>] md_make_request+0xdf/0x240
 [<ffffffff8127ddb0>] generic_make_request+0x240/0x5a0
 [<ffffffff81130ba5>] ? mempool_alloc_slab+0x15/0x20
 [<ffffffff81130d43>] ? mempool_alloc+0x63/0x140
 [<ffffffff8127e180>] submit_bio+0x70/0x120
 [<ffffffff811cfc8d>] submit_bh+0x11d/0x1f0
 [<ffffffff811d2a9c>] block_read_full_page+0x27c/0x3a0
 [<ffffffff811d6320>] ? blkdev_get_block+0x0/0x20
 [<ffffffff811513ee>] ? __inc_zone_page_state+0x2e/0x30
 [<ffffffff81145480>] ? __lru_cache_add+0x40/0x90
 [<ffffffff811d7498>] blkdev_readpage+0x18/0x20
 [<ffffffff811441aa>] __do_page_cache_readahead+0x20a/0x210
 [<ffffffff81144251>] force_page_cache_readahead+0x71/0xa0
 [<ffffffff81144773>] page_cache_sync_readahead+0x43/0x50
 [<ffffffff81130128>] generic_file_aio_read+0x558/0x700
 [<ffffffff811d6891>] blkdev_aio_read+0x51/0x80
 [<ffffffff81199a6a>] do_sync_read+0xfa/0x140
 [<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff8115f3d0>] ? unmap_region+0x110/0x130
 [<ffffffff8123aa06>] ? security_file_permission+0x16/0x20
 [<ffffffff8119a365>] vfs_read+0xb5/0x1a0
 [<ffffffff8119b116>] ? fget_light_pos+0x16/0x50
 [<ffffffff8119a6b1>] sys_read+0x51/0xb0
 [<ffffffff8100b0d2>] system_call_fastpath+0x16/0x1b
INFO: task mdadm:2357 blocked for more than 120 seconds.
      Not tainted 2.6.32-642.4.2.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
mdadm         D 0000000000000002     0  2357   2179 0x00000004
 ffff880821773688 0000000000000082 0000000000000000 0000000000000086
 ffff880821773618 ffffffff810633a3 0000013e9de36948 ffff88081f96d440
 ffff88081dc65ed0 000000010010516a ffff88081d713068 ffff880821773fd8
Call Trace:
 [<ffffffff810633a3>] ? __wake_up+0x53/0x70
 [<ffffffffa0240805>] get_active_stripe+0x2d5/0x880 [raid456]
 [<ffffffff81130d00>] ? mempool_alloc+0x20/0x140
 [<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff810a6bce>] ? prepare_to_wait+0x4e/0x80
 [<ffffffffa02455ef>] make_request+0x19f/0xcb0 [raid456]
 [<ffffffff812ab7de>] ? __sg_alloc_table+0x7e/0x130
 [<ffffffff81056cc5>] ? gup_pte_range+0xe5/0x130
 [<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff8143066f>] md_make_request+0xdf/0x240
 [<ffffffff8127ddb0>] generic_make_request+0x240/0x5a0
 [<ffffffff811d941c>] ? do_direct_IO+0x57c/0xfa0
 [<ffffffff811d60cb>] ? bio_alloc_bioset+0x5b/0xf0
 [<ffffffff8127e180>] submit_bio+0x70/0x120
 [<ffffffff811daabd>] __blockdev_direct_IO_newtrunc+0xc7d/0x1270
 [<ffffffff811d6320>] ? blkdev_get_block+0x0/0x20
 [<ffffffff811db127>] __blockdev_direct_IO+0x77/0xe0
 [<ffffffff811d6320>] ? blkdev_get_block+0x0/0x20
 [<ffffffff811d73a7>] blkdev_direct_IO+0x57/0x60
 [<ffffffff811d6320>] ? blkdev_get_block+0x0/0x20
 [<ffffffff8113028b>] generic_file_aio_read+0x6bb/0x700
 [<ffffffff8143877f>] ? md_ioctl+0x31f/0x1ac0
 [<ffffffff811d7f00>] ? blkdev_open+0x0/0xc0
 [<ffffffff81196d47>] ? __dentry_open+0x257/0x380
 [<ffffffff811d6891>] blkdev_aio_read+0x51/0x80
 [<ffffffff81199a6a>] do_sync_read+0xfa/0x140
 [<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff811d66bc>] ? block_ioctl+0x3c/0x40
 [<ffffffff811af522>] ? vfs_ioctl+0x22/0xa0
 [<ffffffff811af6c4>] ? do_vfs_ioctl+0x84/0x580
 [<ffffffff8123aa06>] ? security_file_permission+0x16/0x20
 [<ffffffff8119a365>] vfs_read+0xb5/0x1a0
 [<ffffffff8119b116>] ? fget_light_pos+0x16/0x50
 [<ffffffff8119a6b1>] sys_read+0x51/0xb0
 [<ffffffff8100b0d2>] system_call_fastpath+0x16/0x1b
INFO: task md128_reshape:2428 blocked for more than 120 seconds.
      Not tainted 2.6.32-642.4.2.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
md128_reshape D 0000000000000001     0  2428      2 0x00000000
 ffff88081fdcfa60 0000000000000046 0000000000000000 0000000000000086
 ffff88081fdcf9f0 ffffffff810633a3 0000013d983220e0 ffff88081f96d440
 ffff88081dc65ed0 0000000100103f9f ffff88081ea1e5f8 ffff88081fdcffd8
Call Trace:
 [<ffffffff810633a3>] ? __wake_up+0x53/0x70
 [<ffffffffa0240805>] get_active_stripe+0x2d5/0x880 [raid456]
 [<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffffa02447c0>] reshape_request+0x440/0xa20 [raid456]
 [<ffffffff810633a3>] ? __wake_up+0x53/0x70
 [<ffffffffa02450b2>] sync_request+0x312/0x3a0 [raid456]
 [<ffffffff81430ff7>] md_do_sync+0x6c7/0xd60
 [<ffffffff81431ae5>] md_thread+0x115/0x150
 [<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff814319d0>] ? md_thread+0x0/0x150
 [<ffffffff810a640e>] kthread+0x9e/0xc0
 [<ffffffff8100c28a>] child_rip+0xa/0x20
 [<ffffffff810a6370>] ? kthread+0x0/0xc0
 [<ffffffff8100c280>] ? child_rip+0x0/0x20
INFO: task blkid:2430 blocked for more than 120 seconds.
      Not tainted 2.6.32-642.4.2.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
blkid         D 0000000000000002     0  2430      1 0x00000000
 ffff8808211d36f8 0000000000000082 0000000000000000 0000000000000082
 ffff8808211d3688 ffffffff810633a3 0000013d9831af63 ffff88081f96d440
 ffff88081dc65ed0 0000000100104045 ffff88081e615ad8 ffff8808211d3fd8
Call Trace:
 [<ffffffff810633a3>] ? __wake_up+0x53/0x70
 [<ffffffffa0240805>] get_active_stripe+0x2d5/0x880 [raid456]
 [<ffffffff8142af00>] ? try_module_get+0x30/0xb0
 [<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff810a6bce>] ? prepare_to_wait+0x4e/0x80
 [<ffffffffa02455ef>] make_request+0x19f/0xcb0 [raid456]
 [<ffffffff81340b4b>] ? mix_pool_bytes_extract+0x16b/0x180
 [<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff81151319>] ? zone_statistics+0x99/0xc0
 [<ffffffff8143066f>] md_make_request+0xdf/0x240
 [<ffffffff8127ddb0>] generic_make_request+0x240/0x5a0
 [<ffffffff81130ba5>] ? mempool_alloc_slab+0x15/0x20
 [<ffffffff81130d43>] ? mempool_alloc+0x63/0x140
 [<ffffffff8127e180>] submit_bio+0x70/0x120
 [<ffffffff811cfc8d>] submit_bh+0x11d/0x1f0
 [<ffffffff811d2a9c>] block_read_full_page+0x27c/0x3a0
 [<ffffffff811d6320>] ? blkdev_get_block+0x0/0x20
 [<ffffffff811513ee>] ? __inc_zone_page_state+0x2e/0x30
 [<ffffffff81145480>] ? __lru_cache_add+0x40/0x90
 [<ffffffff811d7498>] blkdev_readpage+0x18/0x20
 [<ffffffff811441aa>] __do_page_cache_readahead+0x20a/0x210
 [<ffffffff81144251>] force_page_cache_readahead+0x71/0xa0
 [<ffffffff81144773>] page_cache_sync_readahead+0x43/0x50
 [<ffffffff81130128>] generic_file_aio_read+0x558/0x700
 [<ffffffff811d6891>] blkdev_aio_read+0x51/0x80
 [<ffffffff81199a6a>] do_sync_read+0xfa/0x140
 [<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff8115f3d0>] ? unmap_region+0x110/0x130
 [<ffffffff8123aa06>] ? security_file_permission+0x16/0x20
 [<ffffffff8119a365>] vfs_read+0xb5/0x1a0
 [<ffffffff8119b116>] ? fget_light_pos+0x16/0x50
 [<ffffffff8119a6b1>] sys_read+0x51/0xb0
 [<ffffffff8100b0d2>] system_call_fastpath+0x16/0x1b
INFO: task mdadm:2481 blocked for more than 120 seconds.
      Not tainted 2.6.32-642.4.2.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
mdadm         D 0000000000000003     0  2481   2459 0x00000004
 ffff8808226ef688 0000000000000086 0000000000000000 0000000000000086
 ffff8808226ef618 ffffffff810633a3 0000014fb75098d5 ffff88081f96d440
 ffff88081dc65ed0 0000000100117098 ffff88081e7f7068 ffff8808226effd8
Call Trace:
 [<ffffffff810633a3>] ? __wake_up+0x53/0x70
 [<ffffffffa0240805>] get_active_stripe+0x2d5/0x880 [raid456]
 [<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff810a6bce>] ? prepare_to_wait+0x4e/0x80
 [<ffffffffa02455ef>] make_request+0x19f/0xcb0 [raid456]
 [<ffffffff8117ec7c>] ? transfer_objects+0x5c/0x80
 [<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff8117f66e>] ? cache_alloc_refill+0x9e/0x240
 [<ffffffff8143066f>] md_make_request+0xdf/0x240
 [<ffffffff8127ddb0>] generic_make_request+0x240/0x5a0
 [<ffffffff811d941c>] ? do_direct_IO+0x57c/0xfa0
 [<ffffffff81151319>] ? zone_statistics+0x99/0xc0
 [<ffffffff811d60cb>] ? bio_alloc_bioset+0x5b/0xf0
 [<ffffffff8127e180>] submit_bio+0x70/0x120
 [<ffffffff811daabd>] __blockdev_direct_IO_newtrunc+0xc7d/0x1270
 [<ffffffff811d6320>] ? blkdev_get_block+0x0/0x20
 [<ffffffff811db127>] __blockdev_direct_IO+0x77/0xe0
 [<ffffffff811d6320>] ? blkdev_get_block+0x0/0x20
 [<ffffffff811d73a7>] blkdev_direct_IO+0x57/0x60
 [<ffffffff811d6320>] ? blkdev_get_block+0x0/0x20
 [<ffffffff8113028b>] generic_file_aio_read+0x6bb/0x700
 [<ffffffff8143877f>] ? md_ioctl+0x31f/0x1ac0
 [<ffffffff811d7f00>] ? blkdev_open+0x0/0xc0
 [<ffffffff81196d47>] ? __dentry_open+0x257/0x380
 [<ffffffff811d6891>] blkdev_aio_read+0x51/0x80
 [<ffffffff81199a6a>] do_sync_read+0xfa/0x140
 [<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff811d66bc>] ? block_ioctl+0x3c/0x40
 [<ffffffff811af522>] ? vfs_ioctl+0x22/0xa0
 [<ffffffff811af6c4>] ? do_vfs_ioctl+0x84/0x580
 [<ffffffff8123aa06>] ? security_file_permission+0x16/0x20
 [<ffffffff8119a365>] vfs_read+0xb5/0x1a0
 [<ffffffff8119b116>] ? fget_light_pos+0x16/0x50
 [<ffffffff8119a6b1>] sys_read+0x51/0xb0
 [<ffffffff8100b0d2>] system_call_fastpath+0x16/0x1b
INFO: task pvscan:2518 blocked for more than 120 seconds.
      Not tainted 2.6.32-642.4.2.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
pvscan        D 0000000000000003     0  2518   2505 0x00000000
 ffff8808205e3688 0000000000000086 0000000000000000 0000000000000086
 ffff8808205e3618 ffffffff810633a3 000001550b5d4884 ffff88081f96d440
 ffff88081dc65ed0 000000010011ca13 ffff88081ddf5ad8 ffff8808205e3fd8
Call Trace:
 [<ffffffff810633a3>] ? __wake_up+0x53/0x70
 [<ffffffffa0240805>] get_active_stripe+0x2d5/0x880 [raid456]
 [<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff810a6bce>] ? prepare_to_wait+0x4e/0x80
 [<ffffffffa02455ef>] make_request+0x19f/0xcb0 [raid456]
 [<ffffffff812ab7de>] ? __sg_alloc_table+0x7e/0x130
 [<ffffffff81056cc5>] ? gup_pte_range+0xe5/0x130
 [<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff8143066f>] md_make_request+0xdf/0x240
 [<ffffffff8127ddb0>] generic_make_request+0x240/0x5a0
 [<ffffffff811d941c>] ? do_direct_IO+0x57c/0xfa0
 [<ffffffff811d60cb>] ? bio_alloc_bioset+0x5b/0xf0
 [<ffffffff8127e180>] submit_bio+0x70/0x120
 [<ffffffff811daabd>] __blockdev_direct_IO_newtrunc+0xc7d/0x1270
 [<ffffffff811d6320>] ? blkdev_get_block+0x0/0x20
 [<ffffffff811db127>] __blockdev_direct_IO+0x77/0xe0
 [<ffffffff811d6320>] ? blkdev_get_block+0x0/0x20
 [<ffffffff811d73a7>] blkdev_direct_IO+0x57/0x60
 [<ffffffff811d6320>] ? blkdev_get_block+0x0/0x20
 [<ffffffff8113028b>] generic_file_aio_read+0x6bb/0x700
 [<ffffffff811d7ef0>] ? blkdev_get+0x10/0x20
 [<ffffffff811d7f00>] ? blkdev_open+0x0/0xc0
 [<ffffffff81196d47>] ? __dentry_open+0x257/0x380
 [<ffffffff811d6891>] blkdev_aio_read+0x51/0x80
 [<ffffffff81199a6a>] do_sync_read+0xfa/0x140
 [<ffffffff81397a3f>] ? scsi_device_put+0x2f/0x40
 [<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff811d66bc>] ? block_ioctl+0x3c/0x40
 [<ffffffff811af522>] ? vfs_ioctl+0x22/0xa0
 [<ffffffff811af6c4>] ? do_vfs_ioctl+0x84/0x580
 [<ffffffff8123aa06>] ? security_file_permission+0x16/0x20
 [<ffffffff8119a365>] vfs_read+0xb5/0x1a0
 [<ffffffff8119b116>] ? fget_light_pos+0x16/0x50
 [<ffffffff8119a6b1>] sys_read+0x51/0xb0
 [<ffffffff8100b0d2>] system_call_fastpath+0x16/0x1b
INFO: task mdadm:2357 blocked for more than 120 seconds.
      Not tainted 2.6.32-642.4.2.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
mdadm         D 0000000000000002     0  2357   2179 0x00000004
 ffff880821773688 0000000000000082 0000000000000000 0000000000000086
 ffff880821773618 ffffffff810633a3 0000013e9de36948 ffff88081f96d440
 ffff88081dc65ed0 000000010010516a ffff88081d713068 ffff880821773fd8
Call Trace:
 [<ffffffff810633a3>] ? __wake_up+0x53/0x70
 [<ffffffffa0240805>] get_active_stripe+0x2d5/0x880 [raid456]
 [<ffffffff81130d00>] ? mempool_alloc+0x20/0x140
 [<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff810a6bce>] ? prepare_to_wait+0x4e/0x80
 [<ffffffffa02455ef>] make_request+0x19f/0xcb0 [raid456]
 [<ffffffff812ab7de>] ? __sg_alloc_table+0x7e/0x130
 [<ffffffff81056cc5>] ? gup_pte_range+0xe5/0x130
 [<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff8143066f>] md_make_request+0xdf/0x240
 [<ffffffff8127ddb0>] generic_make_request+0x240/0x5a0
 [<ffffffff811d941c>] ? do_direct_IO+0x57c/0xfa0
 [<ffffffff811d60cb>] ? bio_alloc_bioset+0x5b/0xf0
 [<ffffffff8127e180>] submit_bio+0x70/0x120
 [<ffffffff811daabd>] __blockdev_direct_IO_newtrunc+0xc7d/0x1270
 [<ffffffff811d6320>] ? blkdev_get_block+0x0/0x20
 [<ffffffff811db127>] __blockdev_direct_IO+0x77/0xe0
 [<ffffffff811d6320>] ? blkdev_get_block+0x0/0x20
 [<ffffffff811d73a7>] blkdev_direct_IO+0x57/0x60
 [<ffffffff811d6320>] ? blkdev_get_block+0x0/0x20
 [<ffffffff8113028b>] generic_file_aio_read+0x6bb/0x700
 [<ffffffff8143877f>] ? md_ioctl+0x31f/0x1ac0
 [<ffffffff811d7f00>] ? blkdev_open+0x0/0xc0
 [<ffffffff81196d47>] ? __dentry_open+0x257/0x380
 [<ffffffff811d6891>] blkdev_aio_read+0x51/0x80
 [<ffffffff81199a6a>] do_sync_read+0xfa/0x140
 [<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff811d66bc>] ? block_ioctl+0x3c/0x40
 [<ffffffff811af522>] ? vfs_ioctl+0x22/0xa0
 [<ffffffff811af6c4>] ? do_vfs_ioctl+0x84/0x580
 [<ffffffff8123aa06>] ? security_file_permission+0x16/0x20
 [<ffffffff8119a365>] vfs_read+0xb5/0x1a0
 [<ffffffff8119b116>] ? fget_light_pos+0x16/0x50
 [<ffffffff8119a6b1>] sys_read+0x51/0xb0
 [<ffffffff8100b0d2>] system_call_fastpath+0x16/0x1b
INFO: task md128_reshape:2428 blocked for more than 120 seconds.
      Not tainted 2.6.32-642.4.2.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
md128_reshape D 0000000000000001     0  2428      2 0x00000000
 ffff88081fdcfa60 0000000000000046 0000000000000000 0000000000000086
 ffff88081fdcf9f0 ffffffff810633a3 0000013d983220e0 ffff88081f96d440
 ffff88081dc65ed0 0000000100103f9f ffff88081ea1e5f8 ffff88081fdcffd8
Call Trace:
 [<ffffffff810633a3>] ? __wake_up+0x53/0x70
 [<ffffffffa0240805>] get_active_stripe+0x2d5/0x880 [raid456]
 [<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffffa02447c0>] reshape_request+0x440/0xa20 [raid456]
 [<ffffffff810633a3>] ? __wake_up+0x53/0x70
 [<ffffffffa02450b2>] sync_request+0x312/0x3a0 [raid456]
 [<ffffffff81430ff7>] md_do_sync+0x6c7/0xd60
 [<ffffffff81431ae5>] md_thread+0x115/0x150
 [<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff814319d0>] ? md_thread+0x0/0x150
 [<ffffffff810a640e>] kthread+0x9e/0xc0
 [<ffffffff8100c28a>] child_rip+0xa/0x20
 [<ffffffff810a6370>] ? kthread+0x0/0xc0
 [<ffffffff8100c280>] ? child_rip+0x0/0x20

Now, commands like pvscan and other commands that control the md array hang. Essentially the only thing I can do is reboot to get back to a state where I can control the array. Even the reboot hangs toward the end, at which point I have to hard reset.

Anyone have any ideas on how to resolve this? I did try booting to an ubunutu live cd, but when I wanted to assemble the array, the entire thing froze.

  • I bit the bullet and recreated the array with the missing devices and readded them. I'm not sure if this is going to work, since it was in the middle of a reshape...but it's recovering now. – Artagel Sep 26 '16 at 18:50
  • md126 : active raid6 sdl[15] sdn[14] sde[13] sdd[12] sdb[11] sdc[10] sdm[7] sdo[6] sdk[5] sdj[4] sdi[3] sdh[2] sdg[1] sdf[0] 46882646016 blocks super 1.2 level 6, 512k chunk, algorithm 2 [14/12] [UUUUUUUU__UUUU] [>....................] recovery = 0.0% (3446904/3906887168) finish=4980.7min speed=13061K/sec – Artagel Sep 26 '16 at 18:50
0

So in the end, I was able to recover this array by doing the following command:

mdadm --create /dev/md126 --level=6 --raid-devices=14 --name=gigantor:128 /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdk /dev/sdo /dev/sdm missing missing /dev/sdc /dev/sdb /dev/sdd /dev/sde --assume-clean

Then adding the two devices back in.

mdadm --add /dev/md126 /dev/sdn
mdadm --add /dev/md126 /dev/sdl

Afterwards, since my LVM info was also corrupted, I had to recover that:

pvcreate --uuid "kvEA4X-vobg-2Ipz-ITF1-ZhtW-Ewej-6liKVx" --restorefile /etc/lvm/backup/vg_data /dev/md126
vgcfgrestore vg_data
lvchange -ay /dev/vg_data/lvm0

Then I tried to xfs_check the LVM, and it said I needed to run repair and destroy the logs:

xfs_repair -L /dev/mapper/vg_data-lvm0

After this repaired, I could now mount my LVM and my data was intact.

My LVM is repairing.

Personalities : [raid6] [raid5] [raid4]
md126 : active raid6 sdm[14] sdl[7] sdk[15] sdn[6] sdj[5] sdi[4] sdg[2] sde[0] sdf[1] sdh[3] sdc[12] sda[11] sdd[13] sdb[10]
      46882646016 blocks super 1.2 level 6, 512k chunk, algorithm 2 [14/12] [UUUUUUUU__UUUU]
      [==========>..........]  recovery = 53.6% (2097061864/3906887168) finish=2448.0min speed=12321K/sec
      bitmap: 0/30 pages [0KB], 65536KB chunk

md127 : active raid6 sdac[15] sdaa[13] sdz[2] sdab[0] sdq[9] sdy[14] sdx[4] sdr[8] sdp[10] sdo[16] sdv[6] sdu[7] sdw[5] sds[12] sdt[11]
      50789533184 blocks super 1.2 level 6, 512k chunk, algorithm 2 [15/15] [UUUUUUUUUUUUUUU]

unused devices: <none>

What a pain....

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.