1

I have looked and looked for other people with my same problem, but all the questions here seem to be about entire RAIDs disappearing after reboots, whereas I have a problem with just one member drive.

This is a video production machine, and last week (after upgrading from CentOS 7 to Rocky 8) we noticed that video playback was producing visual artifacts on the video. All of the video is stored on the attached RAID.

It is a RAID 60, so two RAID6 each containing 12 1.2TB drives and then those two RAID6s come together in a RAID0. This was something set up by an external company a long time before I started working here but it has been solid in my experience.

Investigating these visual artifacts led me to find that, according to mdadm, one of the drives in one of the RAID6 was marked as "removed". The RAID was still working without this drive, as you'd expect from RAID6, but I suspect that it had something to do with the artifacting we were seeing. smartctl showed that the drive in question was failing, so we ordered up a new drive.

It arrived this morning and from there I followed these instructions on redhat.com to the letter. It spent nearly three hours rebuilding the RAID with the new drive but it seemed to work, RAID was back and I wasn't seeing the artifacting.

However, I rebooted the machine and we were back to square one. It was exactly as when we had started, with one of the RAID6s showing a removed drive. Also when I went to look at Disks, I could see that the drive in question (/dev/sdc) had lost it's partitioning, or at least it says "1.2TB Free Space" rather than "1.2TB Linux RAID Member". Thinking (hoping) maybe it was a fluke, I went through the whole process again this evening and the exact same thing has happened. The only thing I did differently this second time was create a /etc/mdadm.conf file using mdadm --examine --scan >> /etc/mdadm/mdadm.conf as su but didn't seem to make a difference. I have now cleared the file just to start afresh.

I cannot for the life of me figure out what's going on. I'm fairly capable with Linux but I didn't even know mdadm existed before this week so I have been trying to learn on the fly. This production machine needs to go back into action on Tuesday so I'm up against it! I'm going to rebuild the RAID again overnight and start fresh tomorrow. Below is all the outputs I think you might need but please let me know if there's anything else I can provide.

Output of cat /proc/mdstat:

Personalities : [raid6] [raid5] [raid4] [raid0] 
md103 : active raid0 md101[0] md102[1]
      23439351808 blocks super 1.2 512k chunks
      
md102 : active raid6 sdu[6] sdz[11] sdx[9] sdw[8] sdy[10] sdq[2] sdt[5] sdr[3] sdv[7] sds[4] sdo[0] sdp[1]
      11719808000 blocks super 1.2 level 6, 512k chunk, algorithm 2 [12/12] [UUUUUUUUUUUU]
      bitmap: 0/9 pages [0KB], 65536KB chunk

md101 : active raid6 sdk[8] sdh[5] sdg[4] sdl[9] sdf[3] sdi[6] sdj[7] sde[2] sdd[1] sdm[10] sdn[11]
      11719808000 blocks super 1.2 level 6, 512k chunk, algorithm 2 [12/11] [_UUUUUUUUUUU]
      bitmap: 1/9 pages [4KB], 65536KB chunk

Output of mdadm --detail for the RAID6 with the issue:

/dev/md101:
           Version : 1.2
     Creation Time : Tue Jun  8 17:37:23 2021
        Raid Level : raid6
        Array Size : 11719808000 (10.91 TiB 12.00 TB)
     Used Dev Size : 1171980800 (1117.69 GiB 1200.11 GB)
      Raid Devices : 12
     Total Devices : 11
       Persistence : Superblock is persistent

     Intent Bitmap : Internal

       Update Time : Fri Jan 12 19:20:16 2024
             State : clean, degraded 
    Active Devices : 11
   Working Devices : 11
    Failed Devices : 0
     Spare Devices : 0

            Layout : left-symmetric
        Chunk Size : 512K

Consistency Policy : bitmap

              Name : grade1:101
              UUID : 56d9ee6d:3a9ef416:91d3b7ec:0da562b0
            Events : 1036527

    Number   Major   Minor   RaidDevice State
       -       0        0        0      removed
       1       8       48        1      active sync   /dev/sdd
       2       8       64        2      active sync   /dev/sde
       3       8       80        3      active sync   /dev/sdf
       4       8       96        4      active sync   /dev/sdg
       5       8      112        5      active sync   /dev/sdh
       6       8      128        6      active sync   /dev/sdi
       7       8      144        7      active sync   /dev/sdj
       8       8      160        8      active sync   /dev/sdk
       9       8      176        9      active sync   /dev/sdl
      10       8      192       10      active sync   /dev/sdm
      11       8      208       11      active sync   /dev/sdn

This might be overkill but output of fdisk -l which is long because there's just so many drives. sda and sdb are the operating system drives, and the problem drive /dev/sdc looks different because I already ran sgdisk on it as per the redhat.com instructions in prep for readding it:

Disk /dev/sda: 894.3 GiB, 960197124096 bytes, 1875385008 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 18255C5C-FE0C-4ADB-9D13-52560809D652

Device       Start        End    Sectors   Size Type
/dev/sda1     2048    1230847    1228800   600M EFI System
/dev/sda2  1230848    3327999    2097152     1G Linux filesystem
/dev/sda3  3328000 1875384319 1872056320 892.7G Linux LVM


Disk /dev/sdb: 894.3 GiB, 960197124096 bytes, 1875385008 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: D2D8699C-C29B-4C34-B126-3667FA7B794A

Device     Start        End    Sectors   Size Type
/dev/sdb1   2048 1875384319 1875382272 894.3G Linux LVM


Disk /dev/mapper/rl-root: 70 GiB, 75161927680 bytes, 146800640 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/mapper/rl-swap: 4 GiB, 4294967296 bytes, 8388608 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/sdd: 1.1 TiB, 1200243695616 bytes, 2344225968 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/sdc: 1.1 TiB, 1200243695616 bytes, 2344225968 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: B9AB730B-09DD-44FF-BD9E-79502FB2CF5E


Disk /dev/sdh: 1.1 TiB, 1200243695616 bytes, 2344225968 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/sdg: 1.1 TiB, 1200243695616 bytes, 2344225968 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/sde: 1.1 TiB, 1200243695616 bytes, 2344225968 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/sdl: 1.1 TiB, 1200243695616 bytes, 2344225968 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/sdi: 1.1 TiB, 1200243695616 bytes, 2344225968 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/sdm: 1.1 TiB, 1200243695616 bytes, 2344225968 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/sdo: 1.1 TiB, 1200243695616 bytes, 2344225968 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/sdp: 1.1 TiB, 1200243695616 bytes, 2344225968 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/sdk: 1.1 TiB, 1200243695616 bytes, 2344225968 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/sdq: 1.1 TiB, 1200243695616 bytes, 2344225968 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/sdf: 1.1 TiB, 1200243695616 bytes, 2344225968 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/sdn: 1.1 TiB, 1200243695616 bytes, 2344225968 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/sdr: 1.1 TiB, 1200243695616 bytes, 2344225968 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/sdj: 1.1 TiB, 1200243695616 bytes, 2344225968 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/sds: 1.1 TiB, 1200243695616 bytes, 2344225968 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/sdt: 1.1 TiB, 1200243695616 bytes, 2344225968 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/sdu: 1.1 TiB, 1200243695616 bytes, 2344225968 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/sdv: 1.1 TiB, 1200243695616 bytes, 2344225968 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/sdw: 1.1 TiB, 1200243695616 bytes, 2344225968 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/sdx: 1.1 TiB, 1200243695616 bytes, 2344225968 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/sdy: 1.1 TiB, 1200243695616 bytes, 2344225968 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/sdz: 1.1 TiB, 1200243695616 bytes, 2344225968 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/mapper/rl-home: 1.7 TiB, 1839227469824 bytes, 3592241152 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/md101: 10.9 TiB, 12001083392000 bytes, 23439616000 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 524288 bytes / 5242880 bytes


Disk /dev/md102: 10.9 TiB, 12001083392000 bytes, 23439616000 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 524288 bytes / 5242880 bytes


Disk /dev/md103: 21.8 TiB, 24001896251392 bytes, 46878703616 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 524288 bytes / 5242880 bytes
Improve this question
3
  • 1
    Wipe the gpt partition table and don't create a new one (skip the sgdisk step)
    – Chris Davies
    Commented Jan 13 at 9:28
  • @ChrisDavies This worked! If you want to put it as an answer I'm happy to mark it as right, although I think you were pretty much telling me to do the same thing as frostschutz was in their answer I followed the instructions in your link to wipe the gpt partition table.
    – schtocker
    Commented Jan 13 at 17:07
  • 1
    Pleased you've fixed it. Suggest you give the answer to frostschutz as they nailed the issue. Any of the three of us can maybe amend the answer to provide the detail you needed
    – Chris Davies
    Commented Jan 13 at 18:08

1 Answer 1

Reset to default
2

Something is wiping your mdadm metadata. It's probably seeing the GPT backup header at the end of disk, and helpfully "repairing" it by rewriting it at start of disk, wiping your mdadm metadata in the process.

This is a classic problem when using whole drives instead of partitions for RAID, LUKS, filesystems, etc. It works fine until it doesn't because lots of programs try to help you partition your drives. And this can kick not just one drive out of your array, but all of them...

You could attempt to clear the partition table GPT headers (both start and end of disk) using wipefs and then hope that nothing will ever attempt to write a new partition table again.

I prefer to have a partition table and use partitions instead of whole disks (like the setup described in the tutorial you linked). It's more standard / less prone to such accidents, since most software knows to leave partitions alone which can't be said for "unpartitioned" drives.

But in your case it would involve migrating your entire setup which probably isn't what you want, either.

Make backups, in particular metadata / header backups, which includes drive serial numbers, so you know how to assign them later. This might aid you in recovery in case you have a partition table accident like this in the future.

Improve this answer
1
  • 1
    This was the solution. I used the second answer given here from @ChrisDavies comment that uses gdisk to wipe the GPT partition table. For anyone looking at this in the future, when gdisk asks you to wipe MBR as well I also selected yes. Plus, I did this after the drive had been added and resynced to the array and it did not need re-adding after this process, it just worked.
    – schtocker
    Commented Jan 14 at 0:26

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .