0

Somehow, the two drives in my RAID 1 configuration have become attached two devices (md0,md127):

~# lsblk -o NAME,SIZE,MOUNTPOINT,STATE,FSTYPE,MODEL,SERIAL,UUID,LABEL
NAME          SIZE MOUNTPOINT STATE   FSTYPE            MODEL            SERIAL     UUID                                 LABEL
sda           1.8T            running linux_raid_member ST32000542AS                69c415bb-716b-8e0b-b03d-721888a9cb05 turris:0
`-md0         1.8T                    btrfs                                         47039540-3842-4b2b-be2b-b8f76e88189b 
sdb           1.8T            running linux_raid_member ST32000542AS                69c415bb-716b-8e0b-b03d-721888a9cb05 turris:0
`-md127       1.8T /mnt/raid          btrfs                                         47039540-3842-4b2b-be2b-b8f76e88189b 

Why did this happen? How do I get them back to a single device (md0)?

** edit **

mdadm -E /dev/sda:

~# mdadm -E /dev/sda
/dev/sda:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 69c415bb:716b8e0b:b03d7218:88a9cb05
           Name : turris:0  (local to host turris)
  Creation Time : Sun Jul 23 11:52:07 2017
     Raid Level : raid1
   Raid Devices : 2

 Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
     Array Size : 1953383360 (1862.89 GiB 2000.26 GB)
  Used Dev Size : 3906766720 (1862.89 GiB 2000.26 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 4ed3485a:ce6205f4:ecd1f9d0:6e4fb2b5

    Update Time : Wed Oct 11 21:18:53 2017
       Checksum : 8a845e99 - correct
         Events : 623


   Device Role : Active device 0
   Array State : A. ('A' == active, '.' == missing)

mdadm -E /dev/sdb:

~# mdadm -E /dev/sdb
/dev/sdb:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 69c415bb:716b8e0b:b03d7218:88a9cb05
           Name : turris:0  (local to host turris)
  Creation Time : Sun Jul 23 11:52:07 2017
     Raid Level : raid1
   Raid Devices : 2

 Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
     Array Size : 1953383360 (1862.89 GiB 2000.26 GB)
  Used Dev Size : 3906766720 (1862.89 GiB 2000.26 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 7c8a1f7a:184b254c:1b25397c:8162faa4

    Update Time : Wed Oct 11 05:58:52 2017
       Checksum : 9d058b99 - correct
         Events : 345


   Device Role : Active device 1
   Array State : .A ('A' == active, '.' == missing)

mdadm -D /dev/md0:

~# mdadm -D /dev/md0
/dev/md0:
        Version : 1.2
  Creation Time : Sun Jul 23 11:52:07 2017
     Raid Level : raid1
     Array Size : 1953383360 (1862.89 GiB 2000.26 GB)
  Used Dev Size : 1953383360 (1862.89 GiB 2000.26 GB)
   Raid Devices : 2
  Total Devices : 1
    Persistence : Superblock is persistent

    Update Time : Wed Oct 11 21:18:53 2017
          State : clean, degraded 
 Active Devices : 1
Working Devices : 1
 Failed Devices : 0
  Spare Devices : 0

           Name : turris:0  (local to host turris)
           UUID : 69c415bb:716b8e0b:b03d7218:88a9cb05
         Events : 623

    Number   Major   Minor   RaidDevice State
       0       8        0        0      active sync   /dev/sda
       1       0        0        1      removed

mdadm -D /dev/md127:

~# mdadm -D /dev/md127
/dev/md127:
        Version : 1.2
  Creation Time : Sun Jul 23 11:52:07 2017
     Raid Level : raid1
     Array Size : 1953383360 (1862.89 GiB 2000.26 GB)
  Used Dev Size : 1953383360 (1862.89 GiB 2000.26 GB)
   Raid Devices : 2
  Total Devices : 1
    Persistence : Superblock is persistent

    Update Time : Wed Oct 11 05:58:52 2017
          State : clean, degraded 
 Active Devices : 1
Working Devices : 1
 Failed Devices : 0
  Spare Devices : 0

           Name : turris:0  (local to host turris)
           UUID : 69c415bb:716b8e0b:b03d7218:88a9cb05
         Events : 345

    Number   Major   Minor   RaidDevice State
       0       0        0        0      removed
       1       8       16        1      active sync   /dev/sdb

** edit 1 **

Contents of /etc/rc.local:

# Put your custom commands here that should be executed once
# the system init finished. By default this file does nothing.

# Disable NCQ (fix RAID issue)
echo 1 > /sys/block/sda/device/queue_depth
echo 1 > /sys/block/sdb/device/queue_depth
# /fix

# start RAID array
mdadm --assemble --scan

exit 0

Contents of /etc/mdadm/mdadm.conf:

# mdadm.conf
#
# Please refer to mdadm.conf(5) for information about this file.
#

DEVICE partitions

# auto-create devices with Debian standard permissions
CREATE owner=root group=disk mode=0660 auto=yes

# automatically tag new arrays as belonging to the local system
HOMEHOST <system>

# instruct the monitoring daemon where to send mail alerts
MAILADDR root

# definitions of existing MD arrays
ARRAY /dev/md/0 metadata=1.2 name=turris:0 UUID=69c415bb:716b8e0b:b03d7218:88a9cb05

Contents (somewhat redacted) of /etc/aliases:

root: cra***@*****.com
  • Is it a hardware raid (as "md127" suggests) or mdadm based raid? – MLu Oct 12 '17 at 22:49
  • @MLu, the fact that it is assigned the number 127 in no way indicates that it is a hardware raid. That simply means that it is part of an array that is not described in /etc/mdadm.conf. @craig, please add the output of mdadm -E /dev/sda and mdadm -E /dev/sdb to your question, as well as mdadm -D /dev/md0 and mdadm -D /dev/md127. – psusi Oct 12 '17 at 23:51
  • @MLu, it's a mdadm-based RAID. – craig Oct 13 '17 at 0:52
1
Events : 623
Events : 345

At this point, your two drives are sufficiently out of sync that there's no easy way to get them back together. Assuming the event counts accurately reflect the relative age of the data on the drives, I recommend wiping /dev/sdb and re-adding it to /dev/md0:

  1. Make sure you've got an up-to-date backup of your data.
  2. Bring down /dev/md127: umount /mnt/raid, mdadm --stop /dev/md127
  3. Make /dev/sdb no longer look like a RAID member: wipefs -a /dev/sdb (the quick way) or dd if=/dev/zero of=/dev/sdb (the thorough way).
  4. Add it to /dev/md0 as a new device: mdadm --manage /dev/md0 --add /dev/sdb
  5. Wait for the array to rebuild.
  6. While you're waiting, turn on monitoring for failures: nano -w /etc/mdadm.conf and add the line MAILADDR your.email@address.com somewhere towards the end, then activate the mdadm monitoring service (this is distro-specific).
  7. Activate /dev/md0: mdadm --run /dev/md0 (may not be needed) followed by mount /dev/md0 /mnt/raid.

As to what caused it, I'm guessing that at one point you had a transient failure of /dev/sdb (Seagate drives tend to do this) and it fell out of the array until the next time you restarted your computer. Since the event counts differed, mdadm wasn't able to put the two drives into a single RAID-1 array, and instead decided to create two single-drive RAID-1 arrays.

  • Yes, it's possible to perform step 7 immediately after step 4, but if you're running RAID, you really should be monitoring its status as well. – Mark Oct 16 '17 at 22:33
  • I was using mdadm --assemble --scan in the /etc/rc.local file to start the array. Will creating the /etc/mdadm/mdadm.conf file supplant the need to do this? – craig Oct 25 '17 at 19:29
  • You'll still need to run mdadm to get it to assemble the array. A properly-configured mdadm.conf simply makes the process more controllable. – Mark Oct 25 '17 at 21:39
  • Is there a way to test the MAILADDR property of the mdadm.conf file to ensure that it is working as expected? – craig Oct 26 '17 at 2:09
  • 1
    mdadm --monitor --scan --test --oneshot will send an email reporting a TestMessage event for each RAID array on your system. – Mark Oct 26 '17 at 4:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.