0

I'm hoping someone can steer me here with a re-build/sync issue. I have a 3 raid 5's build in mdadm 4 disks in each md0 (Active, degraded), md1 (Active, degraded) and md2 (Clean) -no data on it, so I don't care about it.

All 12 drives are attached to an LSI 9750-8i card. The dives have all been attached to this card as single 'units'. One drive failed and showed as missing from the LSI 3dm2 interface. Within Mdadm, the array also shows the drive as 'removed'.

To complicate things further, the second array also shows as Active, degraded with a dive 'removed'

I've shutdown the system and replaced the failed 3TB drive with another new drive. Since I know which drives were present before the failure, I know which two drives are no longer apart of the arrays. sdh = drive that has jumped out of one of the arrays. sd0 = the new physical drive that has been installed. The Superblocks are persistent.

Problem #1. I can query the dives for their serial numbers but I can't seem to find out which array sdh was originally apart of, with the hope I can add it back to the correct array.

Problem #2. add sdo into the md'x' array depending on which one sdf belonged to.

There were some physical drive errors reported within LSI 3dm2 but after re-scanning these disappeared and all the drives at this level are now reporting 'OK'. However, this may have been part of the problem for the sdf to 'fall out' of the array initially?

Sdo I assume, I'll have to use gpart to format the 3TB drive and then add it to the other array? I have backups for this is production NAS. I'd prefer not to hack at it an make a mistake causing a rebuild...

If someone can make some suggestions on how to go about it that would be great! TIA

From cat /proc

  [root@pithos dev]# cat /proc/mdstat   Personalities : [raid6] [raid5 [raid4]
   md0 : active raid5 sdc1[0] sdd1[4] sde1[2]
        8789025216 blocks super 1.2 level 5, 64k chunk, algorithm 2 [4/3] [U_UU]

   md1 : active raid5 sdf1[0] sdi1[4] sdg1[1]
       8789025216 blocks super 1.2 level 5, 64k chunk, algorithm 2 [4/3] [UU_U]

   md2 : active raid5 sdj1[0] sdm1[4] sdl1[2] sdk1[1]
       8789025216 blocks super 1.2 level 5, 64k chunk, algorithm 2 [4/4] [UUUU]

 unused devices: <none>

Md0

[root@pithos dev]# mdadm --detail /dev/md0
/dev/md0:
        Version : 1.2
  Creation Time : Fri May 31 10:46:54 2013
     Raid Level : raid5
     Array Size : 8789025216 (8381.87 GiB 8999.96 GB)
  Used Dev Size : 2929675072 (2793.96 GiB 2999.99 GB)
   Raid Devices : 4
  Total Devices : 3
    Persistence : Superblock is persistent

    Update Time : Fri Jul  8 18:07:27 2016
          State : active, degraded
 Active Devices : 3
Working Devices : 3
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 64K

           Name : pithos.dougallmedia.com:0  (local to host pithos.dougallmedia.com)
           UUID : f1e42a86:7fc791f8:7cf63ab8:065ffa92
         Events : 3543

    Number   Major   Minor   RaidDevice State
       0       8       33        0      active sync   /dev/sdc1
       1       0        0        1      removed
       2       8       65        2      active sync   /dev/sde1
       4       8       49        3      active sync   /dev/sdd1

Md1

[root@pithos dev]# mdadm -D /dev/md1
/dev/md1:
        Version : 1.2
  Creation Time : Fri May 31 17:44:49 2013
     Raid Level : raid5
     Array Size : 8789025216 (8381.87 GiB 8999.96 GB)
  Used Dev Size : 2929675072 (2793.96 GiB 2999.99 GB)
   Raid Devices : 4
  Total Devices : 3
    Persistence : Superblock is persistent

    Update Time : Tue Jul 12 11:31:52 2016
          State : clean, degraded
 Active Devices : 3
Working Devices : 3
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 64K

           Name : pithos.dougallmedia.com:1  (local to host pithos.dougallmedia.com)
           UUID : 58ba8949:bed7be0b:ba4dcc3e:8e6bc5e2
         Events : 9779

    Number   Major   Minor   RaidDevice State
       0       8       81        0      active sync   /dev/sdf1
       1       8       97        1      active sync   /dev/sdg1
       2       0        0        2      removed
       4       8      129        3      active sync   /dev/sdi1


[root@pithos dev]# mdadm --assemble --scan -v
mdadm: looking for devices for /dev/md2
mdadm: Cannot assemble mbr metadata on /dev/sdo
mdadm: cannot open device /dev/dm-10: Device or resource busy
mdadm: cannot open device /dev/dm-9: Device or resource busy
mdadm: cannot open device /dev/dm-8: Device or resource busy
mdadm: cannot open device /dev/dm-7: Device or resource busy
mdadm: cannot open device /dev/dm-6: Device or resource busy
mdadm: cannot open device /dev/dm-5: Device or resource busy
mdadm: cannot open device /dev/md0: Device or resource busy
mdadm: cannot open device /dev/md1: Device or resource busy
mdadm: no recogniseable superblock on /dev/md2
mdadm: Cannot assemble mbr metadata on /dev/sdn
mdadm: cannot open device /dev/dm-4: Device or resource busy
mdadm: cannot open device /dev/dm-3: Device or resource busy
mdadm: cannot open device /dev/dm-2: Device or resource busy
mdadm: cannot open device /dev/dm-1: Device or resource busy
mdadm: cannot open device /dev/dm-0: Device or resource busy
mdadm: cannot open device /dev/sdk1: Device or resource busy
mdadm: cannot open device /dev/sdk: Device or resource busy
mdadm: cannot open device /dev/sdm1: Device or resource busy
mdadm: cannot open device /dev/sdm: Device or resource busy
mdadm: cannot open device /dev/sdl1: Device or resource busy
mdadm: cannot open device /dev/sdl: Device or resource busy
mdadm: cannot open device /dev/sdj1: Device or resource busy
mdadm: cannot open device /dev/sdj: Device or resource busy
mdadm: cannot open device /dev/sdg1: Device or resource busy
mdadm: cannot open device /dev/sdg: Device or resource busy
mdadm: cannot open device /dev/sdi1: Device or resource busy
mdadm: cannot open device /dev/sdi: Device or resource busy
mdadm: /dev/sdh1 has wrong uuid.
mdadm: Cannot assemble mbr metadata on /dev/sdh
mdadm: cannot open device /dev/sdd1: Device or resource busy
mdadm: cannot open device /dev/sdd: Device or resource busy
mdadm: cannot open device /dev/sde1: Device or resource busy
mdadm: cannot open device /dev/sde: Device or resource busy
mdadm: cannot open device /dev/sdf1: Device or resource busy
mdadm: cannot open device /dev/sdf: Device or resource busy
mdadm: cannot open device /dev/sdc1: Device or resource busy
mdadm: cannot open device /dev/sdc: Device or resource busy
mdadm: cannot open device /dev/sdb: Device or resource busy
mdadm: cannot open device /dev/sda: Device or resource busy
mdadm: looking for devices for /dev/md1
mdadm: Cannot assemble mbr metadata on /dev/sdo
mdadm: cannot open device /dev/dm-10: Device or resource busy
mdadm: cannot open device /dev/dm-9: Device or resource busy
mdadm: cannot open device /dev/dm-8: Device or resource busy
mdadm: cannot open device /dev/dm-7: Device or resource busy
mdadm: cannot open device /dev/dm-6: Device or resource busy
mdadm: cannot open device /dev/dm-5: Device or resource busy
mdadm: cannot open device /dev/md0: Device or resource busy
mdadm: cannot open device /dev/md1: Device or resource busy
mdadm: no recogniseable superblock on /dev/md2
mdadm: Cannot assemble mbr metadata on /dev/sdn
mdadm: cannot open device /dev/dm-4: Device or resource busy
mdadm: cannot open device /dev/dm-3: Device or resource busy
mdadm: cannot open device /dev/dm-2: Device or resource busy
mdadm: cannot open device /dev/dm-1: Device or resource busy
mdadm: cannot open device /dev/dm-0: Device or resource busy
mdadm: cannot open device /dev/sdk1: Device or resource busy
mdadm: cannot open device /dev/sdk: Device or resource busy
mdadm: cannot open device /dev/sdm1: Device or resource busy
mdadm: cannot open device /dev/sdm: Device or resource busy
mdadm: cannot open device /dev/sdl1: Device or resource busy
mdadm: cannot open device /dev/sdl: Device or resource busy
mdadm: cannot open device /dev/sdj1: Device or resource busy
mdadm: cannot open device /dev/sdj: Device or resource busy
mdadm: cannot open device /dev/sdg1: Device or resource busy
mdadm: cannot open device /dev/sdg: Device or resource busy
mdadm: cannot open device /dev/sdi1: Device or resource busy
mdadm: cannot open device /dev/sdi: Device or resource busy
mdadm: /dev/sdh1 has wrong uuid.
mdadm: Cannot assemble mbr metadata on /dev/sdh
mdadm: cannot open device /dev/sdd1: Device or resource busy
mdadm: cannot open device /dev/sdd: Device or resource busy
mdadm: cannot open device /dev/sde1: Device or resource busy
mdadm: cannot open device /dev/sde: Device or resource busy
mdadm: cannot open device /dev/sdf1: Device or resource busy
mdadm: cannot open device /dev/sdf: Device or resource busy
mdadm: cannot open device /dev/sdc1: Device or resource busy
mdadm: cannot open device /dev/sdc: Device or resource busy
mdadm: cannot open device /dev/sdb: Device or resource busy
mdadm: cannot open device /dev/sda: Device or resource busy
mdadm: looking for devices for /dev/md0
mdadm: Cannot assemble mbr metadata on /dev/sdo
mdadm: cannot open device /dev/dm-10: Device or resource busy
mdadm: cannot open device /dev/dm-9: Device or resource busy
mdadm: cannot open device /dev/dm-8: Device or resource busy
mdadm: cannot open device /dev/dm-7: Device or resource busy
mdadm: cannot open device /dev/dm-6: Device or resource busy
mdadm: cannot open device /dev/dm-5: Device or resource busy
mdadm: cannot open device /dev/md0: Device or resource busy
mdadm: cannot open device /dev/md1: Device or resource busy
mdadm: no recogniseable superblock on /dev/md2
mdadm: Cannot assemble mbr metadata on /dev/sdn
mdadm: cannot open device /dev/dm-4: Device or resource busy
mdadm: cannot open device /dev/dm-3: Device or resource busy
mdadm: cannot open device /dev/dm-2: Device or resource busy
mdadm: cannot open device /dev/dm-1: Device or resource busy
mdadm: cannot open device /dev/dm-0: Device or resource busy
mdadm: cannot open device /dev/sdk1: Device or resource busy
mdadm: cannot open device /dev/sdk: Device or resource busy
mdadm: cannot open device /dev/sdm1: Device or resource busy
mdadm: cannot open device /dev/sdm: Device or resource busy
mdadm: cannot open device /dev/sdl1: Device or resource busy
mdadm: cannot open device /dev/sdl: Device or resource busy
mdadm: cannot open device /dev/sdj1: Device or resource busy
mdadm: cannot open device /dev/sdj: Device or resource busy
mdadm: cannot open device /dev/sdg1: Device or resource busy
mdadm: cannot open device /dev/sdg: Device or resource busy
mdadm: cannot open device /dev/sdi1: Device or resource busy
mdadm: cannot open device /dev/sdi: Device or resource busy
mdadm: no recogniseable superblock on /dev/sdh1
mdadm: Cannot assemble mbr metadata on /dev/sdh
mdadm: cannot open device /dev/sdd1: Device or resource busy
mdadm: cannot open device /dev/sdd: Device or resource busy
mdadm: cannot open device /dev/sde1: Device or resource busy
mdadm: cannot open device /dev/sde: Device or resource busy
mdadm: cannot open device /dev/sdf1: Device or resource busy
mdadm: cannot open device /dev/sdf: Device or resource busy
mdadm: cannot open device /dev/sdc1: Device or resource busy
mdadm: cannot open device /dev/sdc: Device or resource busy
mdadm: cannot open device /dev/sdb: Device or resource busy
mdadm: cannot open device /dev/sda: Device or resource busy
Improve this question
6
  • try grepping your old kernel logs for raid|mdadm|sd[a-z]
    – cas
    Jul 13, 2016 at 6:55
  • BTW, as you have recent backups of this NAS, I would very strongly recommend rebuilding it as 6 mirrored pairs using either LVM's RAID-1 (no need for mdadm) one VG made up of 6 RAID-1 arrays, or` btrfs` or even ZFS if you don't mind a non-mainline kernel filesystem. RAID1/RAID-10 is much faster, more robust, much faster to resync, and easier to fix if anything goes wrong. The price is that you get only 50% of storage capacity (n/2) rather than the 75% you're getting now with 3xRAID5 ((n-1)/n). Both btrfs and zfs are also far more flexible about allocating quotas to sub-volumes.
    – cas
    Jul 13, 2016 at 7:52
  • Thanks Cas - duly noted. I'm waiting for another 12 HDD to arrive and yes will be rebuilding to RAID 10 from the LSI card. No more mdadm for me. Now 3 experiences with pooched mdadm arrays I wasn't able to recover from... last night I lost another drive in md0 so I'm pouched now on that one anyway.
    – Bazagee
    Jul 13, 2016 at 13:23
  • There's nothing wrong with mdadm (it's just not necessary if you're also using LVM with raid1/raid-10). I've been using it for years, and still use it now for my / and /boot filesystems (bulk data is on ZFS). mdadm is, IMO, much better than any HW raid with the sole exception of RAID5/6 IFF the RAID card has non-volatile write cache. And IMO RAID5/6 isn't worth using whether you have hardware or software raid. If you've had multiple failures with mdadm, i'd suspect hardware problems.
    – cas
    Jul 13, 2016 at 20:57
  • 1
    Well I pulled the plug on the current config and are setting up 4, Raid 6, 6 drive arrays in hardware. Once that's done I'll allow Openfiler to create a Raid 0 strip across all the arrays which should effectively give us a Raid 60 volume. I do some testing from there.
    – Bazagee
    Jul 14, 2016 at 18:28

1 Answer 1

Reset to default
2

Problem #1. I can query the dives for their serial numbers but I can't seem to find out which array sdh was originally apart of, with the hope I can add it back to the correct array.

Use mdadm -E /dev/sdh1 (examine). Then you can use mdadm --re-add ... or --add.

Also, if your other disks are in good health, you can simply rebuild this disk even though it already had data.

If that doesn't work, you may have to use mdadm --create --assume-clean ... to create a new array on top of the old data, reusing the data, which is dangerous. Be sure to save your detail (-D) and examine output for all disks first. You have to explicitly set many of the things you see in the examine or detail output such as the layout, level, drive order, chunk size, etc. . To use --create safely, be sure to test well with overlays instead of risking your data. Using --assume-clean means it will keep your old data.

Problem #2. add sdo into the md'x' array depending on which one sdf belonged to.

You can do this step after you have used mdadm -E to know which array to add it to, even if you didn't finish with the overlays, and it's also just mdadm --add /dev/md0 /dev/sdo

And also I recommend you run smart long tests on your disks often, and that way you can replace disks before they cause raid issues. (Also Seagate Barracuda 3TB disks are very unreliable... my guess is that is what you have, and it's not mdadm's fault; adding smart tests will help with these types of problems too.)

And I recommend you come to irc.freenode.net #linux-raid for the best realtime help with problems like these.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .