How to replace faulty disk in software raid using lvm

Question

one of the SuSE 12 LinuxServers has reported disk failure. Fortunately the Database Server has Software Raid hence the system is still up and running. But as recommended, we would like to replace the failed disk with a new one and rebuild the software raid on it.

System Information is :

Total 4 Internal Disks. sda, sdb , sdc and sdd

The fdisk partitions are :

fdisk -l 
Disk /dev/sdb: 931.5 GiB, 1000204886016 bytes, 1953525168 sectors 
Units: sectors of 1 * 512 = 512 bytes 
Sector size (logical/physical): 512 bytes / 4096 bytes 
I/O size (minimum/optimal): 4096 bytes / 4096 bytes 
Disklabel type: dos 
Disk identifier: 0x0007d757

Device    Boot   Start           End    Sectors   Size Id Type 
/dev/sdb1 *          2048    2105343    2103296     1G fd Linux raid autodetect 
/dev/sdb2         2105344   39858175   37752832    18G fd Linux raid autodetect 
/dev/sdb3        39858176 1953523711 1913665536 912.5G fd Linux raid autodetect

Disk /dev/sdc: 931.5 GiB, 1000204886016 bytes, 1953525168 sectors
Units: sectors of 1 * 512 = 512 bytes 
Sector size (logical/physical): 512 bytes / 4096 bytes 
I/O size (minimum/optimal): 4096 bytes / 4096 bytes 
Disklabel type: dos 
Disk identifier: 0x000a0e8a

Device    Boot Start        End Sectors      Size Id Type 
/dev/sdc1       2048 1953523711 1953521664 931.5G fd Linux raid autodetect

Disk /dev/sdd: 931.5 GiB, 1000204886016 bytes, 1953525168 sectors
Units: sectors of 1 * 512 = 512 bytes 
Sector size (logical/physical): 512 bytes / 4096 bytes 
I/O size (minimum/optimal): 4096 bytes / 4096 bytes 
Disklabel type: dos 
Disk identifier: 0x000caaad

Device    Boot Start        End    Sectors Size   Id Type 
/dev/sdd1       2048 1953523711 1953521664 931.5G fd Linux raid autodetect

Software RAID --> sda + sdb (sda is failed disk) Software RAID --> sdb + sdc

DBServer# cat /proc/mdstat 
Personalities : [raid1] 

md3 : active raid1 sdc1[0] sdd1[1] 976760640 blocks super 1.0 [2/2] [UU] bitmap: 2/8 pages [8KB], 65536KB chunk

md0 : active raid1 sdb1[1] sda10 1051584 blocks super 1.0 [2/1] [_U] bitmap: 1/1 pages [4KB], 65536KB chunk

md2 : active raid1 sdb3[1] sda30 956832576 blocks super 1.0 [2/1] [_U] bitmap: 2/8 pages [8KB], 65536KB chunk

md1 : active raid1 sdb2[1] sda20 18876288 blocks super 1.0 [2/1] [_U]  bitmap: 0/1 pages [0KB], 65536KB chunk 

unused devices:

So md0,md1 and md2 have failed devices namely sda1,sda2 and sda3

Please note that it also has 2 VGs defined as shown below,

1 VG - system (/dev/md2) 2 VG - ora_db (/dev/md3)

pvdisplay
--- Physical volume --- 
PV Name /dev/md3 
VG Name ora_db 
PV Size 931.51 GiB / not usable 3.81 MiB 
Allocatable yes 
PE Size 4.00 MiB 
Total PE 238466 
Free PE 84866 
Allocated PE 153600 
PV UUID vgPdWQ-x6CW-vvdF-moxh-FKyb-wpSU-NdJqSm

--- Physical volume --- 
PV Name /dev/md2 
VG Name system 
PV Size 912.51 GiB / not usable 2.81 MiB 
Allocatable yes 
PE Size 4.00 MiB 
Total PE 233601 
Free PE 182401 
Allocated PE 51200 
PV UUID rdff2n-ztxd-lcBY-nAqk-8O9u-fnFG-BVI91v

The grub.conf shows : (Relevant part)

if [ x$feature_default_font_path = xy ] ; then
  font=unicode 
else
  insmod part_msdos msdos 
  insmod diskfilter mdraid1x lvm 
  insmod ext2 
  set root='lvmid/m7AEp0-79EG-D2Vi-ELzE-BTzh-C8mN-CLxrpz/S0eZEl-PlBX-E1ZL-oCwL-SmUx-4Qe4-Mz9NHX'
  if [ x$feature_platform_search_hint = xy ]; then 
    search --no-floppy --fs-uuid --set=root --hint='lvmid/m7AEp0-79EG-D2Vi-ELzE-BTzh-C8mN-CLxrpz/S0eZEl-PlBX-E1ZL-oCwL-SmUx-4Qe4-Mz9NHX' 7c2e3a9c-5f5b-47e3-8a0a-d1e66f12747c 
  else 
    search --no-floppy --fs-uuid --set=root 7c2e3a9c-5f5b-47e3-8a0a-d1e66f12747c 
  fi 
  font="/share/grub2/unicode.pf2" 
fi

if loadfont $font ; then 
  set gfxmode=auto 
  load_video 
  insmod gfxterm
  set locale_dir=$prefix/locale 
  set lang=POSIX 
  insmod gettext 
fi
terminal_output gfxterm 
insmod part_msdos msdos 
insmod diskfilter mdraid1x 
insmod ext2 
set root='mduuid/531cd341e2c7d5a71c542ad04d9ea589' 
if [ x$feature_platform_search_hint = xy ]; then 
  search --no-floppy --fs-uuid --set=root --hint='mduuid/531cd341e2c7d5a71c542ad04d9ea589' 96c11697-c3b7-4f11-90fc-3aef207db526 
else 
  search --no-floppy --fs-uuid --set=root 96c11697-c3b7-4f11-90fc-3aef207db526 
fi

Quote: The procedure to follow should go like this,

First we mark /dev/sda1 as failed:

mdadm --manage /dev/md0 --fail /dev/sda1

Then we remove /dev/sda1 from /dev/md0:

mdadm --manage /dev/md0 --remove /dev/sda1

Now we do the same steps again for /dev/sda2 and sda3 (which is part of /dev/md1 and /dev/md2)

Then power down the system:

shutdown -h now 

and replace the old /dev/sdb hard drive with a new one

After inserting new SATA disk /dev/sda, boot the system.

Then we create the exact same partitioning as on /dev/sda. We can do this with one simple command:

sfdisk -d /dev/sdb | sfdisk /dev/sda

Check if both the disks have same partitions (fdisk -l) 8.Next we add /dev/sda1 to /dev/md0 and /dev/sda2 to /dev/md1 and /dev/sda3 to /dev/md3:

mdadm --manage /dev/md0 --add /dev/sda1 mdadm --manage /dev/md1 --add /dev/sda2 mdadm --manage /dev/md2 --add /dev/sda3

Confirm the synchronisation in progress

cat /proc/mdstat

Please let me know if I have missed something. 2 important points I guess would be, how should I take care of lvm and grub in this case.

Do I have to do something extra to take care of it or the command sfdisk -d /dev/sdb | sfdisk /dev/sda , should take care of LVM as well.

How should I take care of grub in this case? As grun.conf shows entries pertaining to LVM as well as MDADM. Do I have to change anything here before I shutdown the system?

I understand the system has 2 pointers to take care of mdadm+lvm. Which have complicated things. Else would it be easier to setup completely new system?

Not yet tried anything. Only collecting information.

grub.cfg

### BEGIN /etc/grub.d/00_header ###
if [ -f ${config_directory}/grubenv ]; then
  load_env -f ${config_directory}/grubenv
elif [ -s $prefix/grubenv ]; then
  load_env
fi

if [ "${env_block}" ] ; then
  load_env -f "${env_block}"
fi

if [ "${next_entry}" ] ; then
   set default="${next_entry}"
   set next_entry=
   save_env next_entry
   if [ "${env_block}" ] ; then
     save_env -f "${env_block}" next_entry
   fi
   set boot_once=true
else
   set default="${saved_entry}"
fi

if [ x"${feature_menuentry_id}" = xy ]; then
  menuentry_id_option="--id"
else
  menuentry_id_option=""
fi

export menuentry_id_option

if [ "${prev_saved_entry}" ]; then
  set saved_entry="${prev_saved_entry}"
  save_env saved_entry
  set prev_saved_entry=
  save_env prev_saved_entry
  set boot_once=true
fi

function savedefault {
  if [ -z "${boot_once}" ]; then
    saved_entry="${chosen}"
    save_env saved_entry
  fi
}

function load_video {
  if [ x$feature_all_video_module = xy ]; then
    insmod all_video
  else
    insmod efi_gop
    insmod efi_uga
    insmod ieee1275_fb
    insmod vbe
    insmod vga
    insmod video_bochs
    insmod video_cirrus
  fi
}

if [ x$feature_default_font_path = xy ] ; then
   font=unicode
else
insmod part_msdos msdos
insmod diskfilter mdraid1x lvm
insmod ext2
set root='lvmid/m7AEp0-79EG-D2Vi-ELzE-BTzh-C8mN-CLxrpz/S0eZEl-PlBX-E1ZL-oCwL-SmUx-4Qe4-Mz9NHX'
if [ x$feature_platform_search_hint = xy ]; then
  search --no-floppy --fs-uuid --set=root --hint='lvmid/m7AEp0-79EG-D2Vi-ELzE-BTzh-C8mN-CLxrpz/S0eZEl-PlBX-E1ZL-oCwL-SmUx-4Qe4-Mz9NHX'  7c2e3a9c-5f5b-47e3-8a0a-d1e66f12747c
else
  search --no-floppy --fs-uuid --set=root 7c2e3a9c-5f5b-47e3-8a0a-d1e66f12747c
fi
    font="/share/grub2/unicode.pf2"
fi

if loadfont $font ; then
  set gfxmode=auto
  load_video
  insmod gfxterm
  set locale_dir=$prefix/locale
  set lang=POSIX
  insmod gettext
fi
terminal_output gfxterm
insmod part_msdos msdos
insmod diskfilter mdraid1x
insmod ext2
set root='mduuid/531cd341e2c7d5a71c542ad04d9ea589'
if [ x$feature_platform_search_hint = xy ]; then
  search --no-floppy --fs-uuid --set=root --hint='mduuid/531cd341e2c7d5a71c542ad04d9ea589'  96c11697-c3b7-4f11-90fc-3aef207db526
else
  search --no-floppy --fs-uuid --set=root 96c11697-c3b7-4f11-90fc-3aef207db526
fi
insmod gfxmenu
loadfont ($root)/grub2/themes/SLE/DejaVuSans-Bold14.pf2
loadfont ($root)/grub2/themes/SLE/DejaVuSans10.pf2
loadfont ($root)/grub2/themes/SLE/DejaVuSans12.pf2
loadfont ($root)/grub2/themes/SLE/ascii.pf2
insmod png
set theme=($root)/grub2/themes/SLE/theme.txt
export theme
if [ x${boot_once} = xtrue ]; then
  set timeout=0
elif [ x$feature_timeout_style = xy ] ; then
  set timeout_style=menu
  set timeout=8
# Fallback normal timeout code in case the timeout_style feature is
# unavailable.
else
  set timeout=8
fi
### END /etc/grub.d/00_header ###

### BEGIN /etc/grub.d/10_linux ###
menuentry 'SLES12'  --class sles12 --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-690785da-f0f0-4250-b693-5a008acbba10' {
        load_video
        set gfxpayload=keep
        insmod gzio
        insmod part_msdos msdos
        insmod diskfilter mdraid1x
        insmod ext2
        set root='mduuid/531cd341e2c7d5a71c542ad04d9ea589'
        if [ x$feature_platform_search_hint = xy ]; then
          search --no-floppy --fs-uuid --set=root --hint='mduuid/531cd341e2c7d5a71c542ad04d9ea589'  96c11697-c3b7-4f11-90fc-3aef207db526
        else
          search --no-floppy --fs-uuid --set=root 96c11697-c3b7-4f11-90fc-3aef207db526
        fi
        echo    'Loading Linux 3.12.28-4-default ...'
        linux   /vmlinuz-3.12.28-4-default root=UUID=690785da-f0f0-4250-b693-5a008acbba10   resume=/dev/md1 splash=silent quiet crashkernel=232M-:116M  showopts
        echo    'Loading initial ramdisk ...'
        initrd  /initrd-3.12.28-4-default
}
submenu 'Advanced options for SLES12' --hotkey=1 $menuentry_id_option 'gnulinux-advanced-690785da-f0f0-4250-b693-5a008acbba10' {
        menuentry 'SLES12, with Linux 3.12.28-4-default' --hotkey=2 --class sles12 --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-3.12.28-4-default-advanced-690785da-f0f0-4250-b693-

5a008acbba10' {
                load_video
                set gfxpayload=keep
                insmod gzio
                insmod part_msdos msdos
                insmod diskfilter mdraid1x
                insmod ext2
                set root='mduuid/531cd341e2c7d5a71c542ad04d9ea589'
                if [ x$feature_platform_search_hint = xy ]; then
                  search --no-floppy --fs-uuid --set=root --hint='mduuid/531cd341e2c7d5a71c542ad04d9ea589'  96c11697-c3b7-4f11-90fc-3aef207db526
                else
                  search --no-floppy --fs-uuid --set=root 96c11697-c3b7-4f11-90fc-3aef207db526
                fi
                echo    'Loading Linux 3.12.28-4-default ...'
                linux   /vmlinuz-3.12.28-4-default root=UUID=690785da-f0f0-4250-b693-5a008acbba10   resume=/dev/md1 splash=silent quiet crashkernel=232M-:116M  showopts
                echo    'Loading initial ramdisk ...'
                initrd  /initrd-3.12.28-4-default
        }
        menuentry 'SLES12, with Linux 3.12.28-4-default (recovery mode)' --hotkey=3 --class sles12 --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-3.12.28-4-default-recovery-

690785da-f0f0-4250-b693-5a008acbba10' {
                load_video
                set gfxpayload=keep
                insmod gzio
                insmod part_msdos msdos
                insmod diskfilter mdraid1x
                insmod ext2
                set root='mduuid/531cd341e2c7d5a71c542ad04d9ea589'
                if [ x$feature_platform_search_hint = xy ]; then
                  search --no-floppy --fs-uuid --set=root --hint='mduuid/531cd341e2c7d5a71c542ad04d9ea589'  96c11697-c3b7-4f11-90fc-3aef207db526
                else
                  search --no-floppy --fs-uuid --set=root 96c11697-c3b7-4f11-90fc-3aef207db526
                fi
                echo    'Loading Linux 3.12.28-4-default ...'
                linux   /vmlinuz-3.12.28-4-default root=UUID=690785da-f0f0-4250-b693-5a008acbba10  showopts apm=off noresume edd=off powersaved=off nohz=off highres=off processor.max_cstate=1 nomodeset 

x11failsafe crashkernel=232M-:116M
                echo    'Loading initial ramdisk ...'
                initrd  /initrd-3.12.28-4-default
        }
}

### END /etc/grub.d/10_linux ###

### BEGIN /etc/grub.d/20_linux_xen ###
### END /etc/grub.d/20_linux_xen ###

### BEGIN /etc/grub.d/20_memtest86+ ###
### END /etc/grub.d/20_memtest86+ ###

### BEGIN /etc/grub.d/30_os-prober ###
### END /etc/grub.d/30_os-prober ###

### BEGIN /etc/grub.d/40_custom ###
# This file provides an easy way to add custom menu entries.  Simply type the
# menu entries you want to add after this comment.  Be careful not to change
# the 'exec tail' line above.
### END /etc/grub.d/40_custom ###

### BEGIN /etc/grub.d/41_custom ###
if [ -f  ${config_directory}/custom.cfg ]; then
  source ${config_directory}/custom.cfg
elif [ -z "${config_directory}" -a -f  $prefix/custom.cfg ]; then
  source $prefix/custom.cfg;
fi
### END /etc/grub.d/41_custom ###

### BEGIN /etc/grub.d/80_suse_btrfs_snapshot ###
### END /etc/grub.d/80_suse_btrfs_snapshot ###

### BEGIN /etc/grub.d/90_persistent ###
### END /etc/grub.d/90_persistent ###

I did install grub on /dev/sdb. Please find the output.

# grub2-install /dev/sdb
Installing for i386-pc platform.
error: cannot read `/dev/sda': Input/output error.
error: cannot read `/dev/sda': Input/output error.
error: cannot read `/dev/sda': Input/output error.
error: cannot read `/dev/sda': Input/output error.
error: cannot read `/dev/sda': Input/output error.
grub2-install: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
error: cannot read `/dev/sda': Input/output error.
error: cannot read `/dev/sda': Input/output error.
error: cannot read `/dev/sda': Input/output error.
error: cannot read `/dev/sda': Input/output error.
error: cannot read `/dev/sda': Input/output error.
grub2-install: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
Installation finished. No error reported.

Not sure why it gave sda errors though.

And then cross checked with following command again.

  # dd bs=512 count=1 if=/dev/sdb 2>/dev/null| strings
    ZRr=
    `|f
    \|f1
    GRUB
    Geom
    Hard Disk
    Read
     Error

Answer 1

LVM is using /dev/md* devices, and the respective /dev/sd* devices are locked by software RAID for exclusive access. While RAID is running, LVM does not care about the /dev/sd* devices that are part of software RAID arrays.

So, as long as both halves of any RAID array do not fail simultaneously, you don't have to do anything at all for LVM.

GRUB, on the other hand, is a different matter. Since software RAID only mirrors what is within the RAID component partitions, the Master Boot Record and the part of GRUB embedded between the MBR and the beginning of the first partition will not be replicated by RAID. However, that'll be easy to fix: just run grub-install /dev/sda and grub-install /dev/sdb to ensure that both copies of your system disk have a valid bootloader.

Your grub.conf snippet includes references to lvmid and mduuid, so it looks like it is already correct for RAID use.

See this question for more info. This is a step you should have already done once when setting up your software mirrored system disk.

However, if you missed this step originally, or if you're even slightly uncertain, it might be wise to run grub-install /dev/sdb right now before rebooting the system, to ensure that the secondary copy of your system disk (which is now the only remaining good copy) in fact has a bootloader on it.

To test whether or not GRUB is installed on your /dev/sdb, you could run this command:

dd if=/dev/sdb bs=512 count=1 2>/dev/null | od -t x1z -A x

If the response indicates that bytes between the beginning of the MBR up to byte 0x1c0 are zeroes, then GRUB is definitely not installed on /dev/sdb.

Example output with no GRUB:

000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  >................<
*
0001c0 02 00 ee fe bf 66 01 00 00 00 ff ff ff ff 00 00  >.....f..........<
0001d0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  >................<
*
0001f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 55 aa  >..............U.<
000200

Note that the first line is all zeroes, and further lines up to 0001c0 are omitted by od (marked with asterisk) since they are just all zeroes too.

If GRUB is installed, you should see no all-zeroes lines before 0001c0, and there should be a string GRUB visible somewhere within the text-mode representation on the right side.

Example output with GRUB installed:

000000 eb 63 90 10 8e d0 bc 00 b0 b8 00 00 8e d8 8e c0  >.c..............<
000010 fb be 00 7c bf 00 06 b9 00 02 f3 a4 ea 21 06 00  >...|.........!..<
000020 00 be be 07 38 04 75 0b 83 c6 10 81 fe fe 07 75  >....8.u........u<
000030 f3 eb 16 b4 02 b0 01 bb 00 7c b2 80 8a 74 01 8b  >.........|...t..<
000040 4c 02 cd 13 ea 00 7c 00 00 eb fe 00 00 00 00 00  >L.....|.........<
000050 00 00 00 00 00 00 00 00 00 00 00 80 01 00 00 00  >................<
000060 00 00 00 00 ff fa 90 90 f6 c2 80 74 05 f6 c2 70  >...........t...p<
000070 74 02 b2 80 ea 79 7c 00 00 31 c0 8e d8 8e d0 bc  >t....y|..1......<
000080 00 20 fb a0 64 7c 3c ff 74 02 88 c2 52 be 80 7d  >. ..d|<.t...R..}<
000090 e8 17 01 be 05 7c b4 41 bb aa 55 cd 13 5a 52 72  >.....|.A..U..ZRr<
0000a0 3d 81 fb 55 aa 75 37 83 e1 01 74 32 31 c0 89 44  >=..U.u7...t21..D<
0000b0 04 40 88 44 ff 89 44 02 c7 04 10 00 66 8b 1e 5c  >[email protected]..\<
0000c0 7c 66 89 5c 08 66 8b 1e 60 7c 66 89 5c 0c c7 44  >|f.\.f..`|f.\..D<
0000d0 06 00 70 b4 42 cd 13 72 05 bb 00 70 eb 76 b4 08  >..p.B..r...p.v..<
0000e0 cd 13 73 0d 5a 84 d2 0f 83 d8 00 be 8b 7d e9 82  >..s.Z........}..<
0000f0 00 66 0f b6 c6 88 64 ff 40 66 89 44 04 0f b6 d1  >[email protected]....<
000100 c1 e2 02 88 e8 88 f4 40 89 44 08 0f b6 c2 c0 e8  >[email protected]......<
000110 02 66 89 04 66 a1 60 7c 66 09 c0 75 4e 66 a1 5c  >.f..f.`|f..uNf.\<
000120 7c 66 31 d2 66 f7 34 88 d1 31 d2 66 f7 74 04 3b  >|f1.f.4..1.f.t.;<
000130 44 08 7d 37 fe c1 88 c5 30 c0 c1 e8 02 08 c1 88  >D.}7....0.......<
000140 d0 5a 88 c6 bb 00 70 8e c3 31 db b8 01 02 cd 13  >.Z....p..1......<
000150 72 1e 8c c3 60 1e b9 00 01 8e db 31 f6 bf 00 80  >r...`......1....<
000160 8e c6 fc f3 a5 1f 61 ff 26 5a 7c be 86 7d eb 03  >......a.&Z|..}..<
000170 be 95 7d e8 34 00 be 9a 7d e8 2e 00 cd 18 eb fe  >..}.4...}.......<
000180 47 52 55 42 20 00 47 65 6f 6d 00 48 61 72 64 20  >GRUB .Geom.Hard <
000190 44 69 73 6b 00 52 65 61 64 00 20 45 72 72 6f 72  >Disk.Read. Error<
0001a0 0d 0a 00 bb 01 00 b4 0e cd 10 ac 3c 00 75 f4 c3  >...........<.u..<
0001b0 00 00 00 00 00 00 00 00 23 c1 00 00 00 00 80 20  >........#...... <
0001c0 21 00 8e fe ff ff 00 08 00 00 00 f8 ff 00 00 00  >!...............<
0001d0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  >................<
*
0001f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 55 aa  >..............U.<
000200

Having a bootloader on a software RAIDed system disk is always tricky, so I would recommend that the administrators of software RAID systems be aware of how to use the rescue boot option of the installation media. Here are the official instructions for rescue boot on SuSE 11 and 12.

In this case, if the system fails to boot after removing the failed disk, you'd need to first perform the procedure in that document, then run grub-install /dev/sda (as the remaining half of the RAID set would probably be detected as /dev/sda after the failed disk was removed).