Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

In short

Following the guide How to Set Up a RAID 1 Under GNU/Linux, I have setup a RAID 1. Checking the RAID Array for validity in various ways, appeared to be normal.

However, after rebooting the system, the RAID Array was not working. The partition in question was not mounted as instructed in /etc/fstab. Assembling the Array manually worked and no data has been lost.

Newly added internal/external disks, which lead to a change of the disk device names (such as a disk being "renamed" from sdd to sde by the system), lead me to accept that it was a problem related with this name-changing fact. This is irrelevant, however, as RAID Arrays are (also) built by using unique UUIDs.

The actual question is why does the Array fail to assemble during the boot process? Or else, what is the boot-script of Funtoo, the operating system under which all of plot takes place, doing regarding the mdadm -assemble process?

The long story

Following the above referenced step by step guide, I have set up a RAID 1 under Funtoo. Checking the RAID 1 Array for validity was done in several ways, mostly using functionalities of the mdadm tool itself.

Specifically, the Array's details were retrieved by using the mdadm tool with the -D flag. The disks that pertain to the Array were examined by using the -E flag. The respective configuration file mdadm.conf can be simply read for if it contains the correct instruction (i.e which md device, what is its UUID and more). Finally, watching the file /proc/mdadm was important to ensure that both disks were active and "synced".

Below follow even more detailed information about the confronted situation.

mdadm -D /dev/md0

/dev/md0:
        Version : 1.2
  Creation Time : Thu Jul 18 00:25:05 2013
     Raid Level : raid1
     Array Size : 1953382208 (1862.89 GiB 2000.26 GB)
  Used Dev Size : 1953382208 (1862.89 GiB 2000.26 GB)
   Raid Devices : 2
  Total Devices : 2
    Persistence : Superblock is persistent

    Update Time : Thu Jul 18 10:33:37 2013
          State : clean 
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0

           Name : Resilience:0  (local to host Resilience)
           UUID : 73bf29ca:89bff887:79a26531:b9733d7a
         Events : 6455

    Number   Major   Minor   RaidDevice State
       2       8       33        0      active sync   /dev/sdc1
       1       8       49        1      active sync   /dev/sdd1

From the command history, I did the following

# trying to follow the guide -- various tests...
      ...
      979  18.Jul.13 [ 00:09:07 ] mdadm --zero-superblock /dev/sdd1
      980  18.Jul.13 [ 00:09:17 ] mdadm --create /dev/md0 --level=1 --raid-disks=2 missing /dev/sdd1
      990  18.Jul.13 [ 00:15:58 ] mdadm --examine --scan

# creating/checking the configuration file

      992  18.Jul.13 [ 00:16:17 ] cat /etc/mdadm.conf 
      993  18.Jul.13 [ 00:16:33 ] mdadm --examine --scan | tee /etc/mdadm.conf
      994  18.Jul.13 [ 00:16:35 ] cat /etc/mdadm.conf 

# after some faulty attempts, finally it goes

      997  18.Jul.13 [ 00:24:45 ] mdadm --stop /dev/md0
      998  18.Jul.13 [ 00:24:54 ] mdadm --zero-superblock /dev/sdd1
      999  18.Jul.13 [ 00:25:04 ] mdadm --create /dev/md0 --level=1 --raid-disks=2 missing /dev/sdd1
     1005  18.Jul.13 [ 00:26:39 ] mdadm --examine --scan | sudo tee /etc/mdadm.conf
     1046  18.Jul.13 [ 03:42:57 ] mdadm --add /dev/md0 /dev/sdc1

The configuration file /etc/mdadm.conf reads:

ARRAY /dev/md/0  metadata=1.2 UUID=73bf29ca:89bff887:79a26531:b9733d7a name=Resilience:0

All works fine as can be seen from /proc/mdadm:

Personalities : [raid6] [raid5] [raid4] [raid1] [raid0] [raid10] [linear] [multipath] 
md0 : active raid1 sdc1[2] sdd1[1]
      1953382208 blocks super 1.2 [2/2] [UU]

unused devices: <none>

Here after, the disks were synced, access was fine. Shutting down the system, adding another disk, external (via USB) or internal, and restarting the system, caused the RAID1 to stop working! The reason is, if I am not wrong, the change of disk device numbers.

In this example, the former sdd1 became sde1 while sdd1 was reserved by a "new" internal disk or an external USB HDD attached before starting the system and instructed to be automatically mounted.

It was very easy to "recover" the "failed" ARRAY by removing all other disks, stopping the Array and re-assembling it. Some of the commands issued while trying and, finally, successfully getting back the ARRAY were:

# booting and unsuccessfully trying to add the "missing" disk

     1091  18.Jul.13 [ 10:22:53 ] mdadm --add /dev/md0 /dev/sdc1
     1092  18.Jul.13 [ 10:28:26 ] mdadm --assemble /dev/md0 --scan
     1093  18.Jul.13 [ 10:28:39 ] mdadm --assemble /dev/md0 --scan --force
     1095  18.Jul.13 [ 10:30:36 ] mdadm --detail /dev/md0

# reading about `mdadm`, trying to "stop", incomplete command though

     1096  18.Jul.13 [ 10:30:45 ] mdadm stop
     1097  18.Jul.13 [ 10:31:12 ] mdadm --examine /dev/sdd
     1098  18.Jul.13 [ 10:31:16 ] mdadm --examine /dev/sdd1
     1099  18.Jul.13 [ 10:31:20 ] mdadm --examine /dev/sdc
     1100  18.Jul.13 [ 10:31:21 ] mdadm --examine /dev/sdc1

# reading again, stop it -- the right way

     1101  18.Jul.13 [ 10:33:19 ] mdadm --stop /dev/md0

# assemble & check

     1102  18.Jul.13 [ 10:33:25 ] mdadm --assemble /dev/md0 --scan
     1111  18.Jul.13 [ 10:34:17 ] mdadm --examine /dev/sd[cd]1

# does the Array have a UUID?

     1112  18.Jul.13 [ 10:37:36 ] UUID=$(mdadm -E /dev/sdd1|perl -ne '/Array UUID : (\S+)/ and print $1')

# below, learning how to report on the Array

     1115  18.Jul.13 [ 10:42:26 ] mdadm -D /dev/md0
     1116  18.Jul.13 [ 10:45:08 ] mdadm --examine /dev/sd[cd]1 >> raid.status
     1197  18.Jul.13 [ 13:16:59 ] mdadm --detail /dev/md0
     1198  18.Jul.13 [ 13:17:29 ] mdadm --examine /dev/sd[cd]1
     1199  18.Jul.13 [ 13:17:41 ] mdadm --help
     1200  18.Jul.13 [ 13:18:41 ] mdadm --monitor /dev/md0
     1201  18.Jul.13 [ 13:18:53 ] mdadm --misc /dev/md0

However, I would expect this not to happen, and be able to play safe as with the rest of disks/partitions when mounting is based on UUIDs and/or LABELs.

The corresponding entry in /etc/fstab reads (note, I skipped the options nosuid,nodev)

/dev/md0        geo     xfs     defaults        0 2

Details for sdc1, output from mdadm -E /dev/sdc1

/dev/sdc1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 73bf29ca:89bff887:79a26531:b9733d7a
           Name : Resilience:0  (local to host Resilience)
  Creation Time : Thu Jul 18 00:25:05 2013
     Raid Level : raid1
   Raid Devices : 2

 Avail Dev Size : 3906764800 (1862.89 GiB 2000.26 GB)
     Array Size : 1953382208 (1862.89 GiB 2000.26 GB)
  Used Dev Size : 3906764416 (1862.89 GiB 2000.26 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 5552ba2d:8d79c88f:c995d052:cef0aa03

    Update Time : Fri Jul 19 11:14:19 2013
       Checksum : 385183dd - correct
         Events : 6455


   Device Role : Active device 0
   Array State : AA ('A' == active, '.' == missing)

Details for the partitions in question

Details for sdd1, output from mdadm -E /dev/sdd1

/dev/sdd1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 73bf29ca:89bff887:79a26531:b9733d7a
           Name : Resilience:0  (local to host Resilience)
  Creation Time : Thu Jul 18 00:25:05 2013
     Raid Level : raid1
   Raid Devices : 2

 Avail Dev Size : 3906764800 (1862.89 GiB 2000.26 GB)
     Array Size : 1953382208 (1862.89 GiB 2000.26 GB)
  Used Dev Size : 3906764416 (1862.89 GiB 2000.26 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 076acfd8:af184e75:f83ce3ae:8e778ba0

    Update Time : Fri Jul 19 11:14:19 2013
       Checksum : c1df68a0 - correct
         Events : 6455


   Device Role : Active device 1
   Array State : AA ('A' == active, '.' == missing)

After adding, again, a "new" internal disk, and rebooting, I experience the same problem.

mdadm -E /dev/sdd1 reports

mdadm: No md superblock detected on /dev/sdd1.

while mdadm -E /dev/sde1

/dev/sde1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 73bf29ca:89bff887:79a26531:b9733d7a
           Name : Resilience:0  (local to host Resilience)
  Creation Time : Thu Jul 18 00:25:05 2013
     Raid Level : raid1
   Raid Devices : 2

 Avail Dev Size : 3906764800 (1862.89 GiB 2000.26 GB)
     Array Size : 1953382208 (1862.89 GiB 2000.26 GB)
  Used Dev Size : 3906764416 (1862.89 GiB 2000.26 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 076acfd8:af184e75:f83ce3ae:8e778ba0

    Update Time : Fri Jul 19 11:34:47 2013
       Checksum : c1df6d6c - correct
         Events : 6455


   Device Role : Active device 1
   Array State : AA ('A' == active, '.' == missing)

and mdadm --detail /dev/md0

mdadm: md device /dev/md0 does not appear to be active.

while cat /proc/mdstat reads

Personalities : [raid6] [raid5] [raid4] [raid1] [raid0] [raid10] [linear] [multipath] 
md0 : inactive sde1[1](S)
      1953382400 blocks super 1.2

unused devices: <none>

Note, as per Gilles' observation, at this point the reported blocks (1953382400) do not match the ones reported for the Array Size : 1953382208, as can be seen above (or below). Obviously, something went wrong here.

The (partial) output of mdadm -Evvvvs is

/dev/sde1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 73bf29ca:89bff887:79a26531:b9733d7a
           Name : Resilience:0  (local to host Resilience)
  Creation Time : Thu Jul 18 00:25:05 2013
     Raid Level : raid1
   Raid Devices : 2

 Avail Dev Size : 3906764800 (1862.89 GiB 2000.26 GB)
     Array Size : 1953382208 (1862.89 GiB 2000.26 GB)
  Used Dev Size : 3906764416 (1862.89 GiB 2000.26 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 076acfd8:af184e75:f83ce3ae:8e778ba0

    Update Time : Fri Jul 19 11:34:47 2013
       Checksum : c1df6d6c - correct
         Events : 6455


   Device Role : Active device 1
   Array State : AA ('A' == active, '.' == missing)
/dev/sde:
   MBR Magic : aa55
Partition[0] :   3907026944 sectors at         2048 (type fd)

/dev/sdc1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 73bf29ca:89bff887:79a26531:b9733d7a
           Name : Resilience:0  (local to host Resilience)
  Creation Time : Thu Jul 18 00:25:05 2013
     Raid Level : raid1
   Raid Devices : 2

 Avail Dev Size : 3906764800 (1862.89 GiB 2000.26 GB)
     Array Size : 1953382208 (1862.89 GiB 2000.26 GB)
  Used Dev Size : 3906764416 (1862.89 GiB 2000.26 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 5552ba2d:8d79c88f:c995d052:cef0aa03

    Update Time : Fri Jul 19 11:34:47 2013
       Checksum : 385188a9 - correct
         Events : 6455


   Device Role : Active device 0
   Array State : AA ('A' == active, '.' == missing)
/dev/sdc:
   MBR Magic : aa55
Partition[0] :   3907026944 sectors at         2048 (type fd)

Checking with fdisk -l, the previous sdc and sdd disks, areis now sdb and sde (recognise them disks from the size among the rest of the drives). It seems that "it" is still looking at/for sdc1sdd1?

Following suggestions found in the comment section, I added more details.

As per derobert's suggestion in the comments, the ARRAY stopped and re-assembled successfully:

# stop it!
mdadm --stop /dev/md0
mdadm: stopped /dev/md0

# re-assemble -- looks good!
mdadm --assemble -v --scan

mdadm: looking for devices for /dev/md/0
mdadm: no RAID superblock on /dev/sdf1
mdadm: no RAID superblock on /dev/sdf
mdadm: no RAID superblock on /dev/sde
mdadm: no RAID superblock on /dev/sdd1
mdadm: no RAID superblock on /dev/sdd
mdadm: no RAID superblock on /dev/sdc
mdadm: no RAID superblock on /dev/sdb
mdadm: no RAID superblock on /dev/sda6
mdadm: no RAID superblock on /dev/sda5
mdadm: no RAID superblock on /dev/sda4
mdadm: no RAID superblock on /dev/sda3
mdadm: no RAID superblock on /dev/sda2
mdadm: no RAID superblock on /dev/sda1
mdadm: no RAID superblock on /dev/sda
mdadm: /dev/sde1 is identified as a member of /dev/md/0, slot 1.
mdadm: /dev/sdc1 is identified as a member of /dev/md/0, slot 0.
mdadm: added /dev/sde1 to /dev/md/0 as 1
mdadm: added /dev/sdc1 to /dev/md/0 as 0
mdadm: /dev/md/0 has been started with 2 drives.

# double-check
mdadm --detail --scan
ARRAY /dev/md/0 metadata=1.2 name=Resilience:0 UUID=73bf29ca:89bff887:79a26531:b9733d7a

New question, how is this to be fixed that without loosing data? As per the discussion and recommendations in the comments, it is related with the boot process? Permissions to the mount point in question perhaps?

mdadm was not registered as a boot service. Adding it and rebooting, did not fix the issue though. A few more, probably interesting, details from dmesg on where it fails:

[   25.356947] md: raid6 personality registered for level 6
[   25.356952] md: raid5 personality registered for level 5
[   25.356955] md: raid4 personality registered for level 4
[   25.383630] md: raid1 personality registered for level 1
[   25.677100] md: raid0 personality registered for level 0
[   26.134282] md: raid10 personality registered for level 10
[   26.257855] md: linear personality registered for level -1
[   26.382152] md: multipath personality registered for level -4
[   41.986222] md: bind<sde1>
[   44.274346] XFS (md0): SB buffer read failed
[   55.028598] ata7: sas eh calling libata cmd error handler
[   55.028615] ata7.00: cmd ef/05:fe:00:00:00/00:00:00:00:00/40 tag 0
[   55.046186] ata7: sas eh calling libata cmd error handler
[   55.046209] ata7.00: cmd ef/c2:00:00:00:00/00:00:00:00:00/40 tag 0
[   55.278378] ata8: sas eh calling libata cmd error handler
[   55.278406] ata8.00: cmd ef/05:fe:00:00:00/00:00:00:00:00/40 tag 0
[   55.349235] ata8: sas eh calling libata cmd error handler
[   55.349290] ata8.00: cmd ef/c2:00:00:00:00/00:00:00:00:00/40 tag 0
[  105.854112] XFS (md0): SB buffer read failed

Further checking the XFS partition in question sde1

xfs_check /dev/sde1
xfs_check: /dev/sde1 is not a valid XFS filesystem (unexpected SB magic number 0x00000000)
xfs_check: WARNING - filesystem uses v1 dirs,limited functionality provided.
xfs_check: read failed: Invalid argument
cache_node_purge: refcount was 1, not zero (node=0x22a64a0)
xfs_check: cannot read root inode (22)
bad superblock magic number 0, giving up

It turns out that the XFS partition in question is not healthy.

An XFS filesystem, although not impossible, is likely not the root of the problem here. Partitions that are part of an RAID Array, are according to Gilles' comment, not an XFS filesystem! The filesystem starts at an offset, there's a RAID header first.

Question(s)

In the beginning

  • Is it possible to "lock" the RAID 1 Array to only work with specific disks/partitions independently from disk device names?

  • Would it be, for example, sufficient to use the Array's UUID in /etc/fstab so as to be immune against changes of the disk device names?

After re-searching the problem

  • At what stage of Funtoo's boot process is the assembmel of the RAID Array attempted? How exactly? Where can it be modified/adjusted?
share|improve this question

closed as off-topic by Michael Mrozek Jul 22 '13 at 4:32

  • This question does not appear to be about Unix or Linux within the scope defined in the help center.
If this question can be reworded to fit the rules in the help center, please edit the question.

    
/etc/fstab isn't considered when assembling the array. Your partition type code should be 0xDA. And please post /etc/mdadm/mdadm.conf. Also, when is the assembly being attempted, during initramfs or after? –  derobert Jul 18 '13 at 16:34
1  
Could you explain exactly what broke, without guessing? Mdraid doesn't use device names, it uses signatures to assemble the array. In other words, /dev/md0 is not built from sdc1 and sdd1, it's built from 73bf29ca:89bff887:79a26531:b9733d7a. The number /dev/md0 is also stable since it comes from /etc/mdadm.conf. However, your bootloader may have been affected. –  Gilles Jul 18 '13 at 23:39
1  
@NikosAlexandris It sounds like you can boot the system with the extra disk attached, and the array doesn't come up. I'd suggest looking at the boot messages and seeing what is different. Also, you can try (without removing the extra disk) mdadm --stop /dev/md0 followed by mdadm --assemble -v --scan. If that puts your array together, then mdadm itself is working, we have to figure out why your OS's boot scripts aren't. If that doesn't put your array together, can we get that output? –  derobert Jul 19 '13 at 0:10
1  
Boot goes in the order 1. Identify disks, create device files for them (/dev/sdX). 2. Assemble arrays, create device files for them (/dev/md0). 3. Mount filesystems. (2) is what is failing, so it'd be very unlikely for a permission issue on the mountpoint to be responsible (as that wouldn't hit until 3). Plus, this is all done as root, so e.g., SELinux, there aren't going to be permission problems. –  derobert Jul 19 '13 at 12:35
1  
/dev/sde1 is not an XFS fileysystem! The filesystem starts at an offset, there's a RAID header first. TO access a RAID volume with a 1.1 or 1.2 signature level, you need to go through mdraid. –  Gilles Jul 19 '13 at 12:38

2 Answers 2

OK—I'm not exactly sure what's going wrong on your box, but how about a quick summary of how this is supposed to work, to aid in troubleshooting.

When you create an array with persistent superblocks (the quite sane default, and the way you've done it), mdadm stores various metadata on each disk. Included in that metadata is the array UUID, the array name, and the slot number. Each disk stores the same UUID and name. The slot number is different for each disk. (If you want to see the metadata, use mdadm -E)

There are three different ways arrays can be assembled on boot:

  1. The kernel can do it. This is what the 0xFD autodetect partition type did. This method discouraged.
  2. Your distro's scripts may do it as part of udev, using incremental assembly, passing the devices to mdadm as they show up. This is the newest method. When it works right, it handles all kinds of asynchronously probed devices like USB, without kluges.
  3. Your distro's scripts call mdadm --assemble --scan after all devices are believed to have been recognized. Kluges are required for, e.g., USB (basically, a sleep to make sure the device nodes have been created). In this mode, mdadm checks (by default) /proc/partitions to find all the block devices on the system, and scans each of them for a superblock.

The distro scripts may do this in the initramfs or later. Or, sometimes, some arrays are done from initramfs (the ones required to get the root filesystem) and others later in the boot.

Regardless of how it's done, mdadm matches devices by looking at the UUID, name, and slot number. It needs to find #0 and #1 for your two-disk array. It really doesn't care which device it is (by default—you can configure it to care in mdadm.conf)

share|improve this answer
    
Quite some information here. Thanks. It means that I should proceed in attaching any disk without hesitation then, given all of current details about the disks and status of the Array (see also updated stuff in the "question") look healthy. Does hardware play any role? Motherboard, ability to hot-swap disks, etc. –  Nikos Alexandris Jul 19 '13 at 8:24
    
@NikosAlexandris Hardware of course plays some role. It has to support hot swap (I believe SATA inherently does, and probably SAS too). In order to actually use it, you'll want hot-swap bays. Next, your controller chip needs to properly signal swap events (modern ones pretty much all do). Finally, the kernel drivers need to support hot swap as well (most do). If all that is in place, then hot swap should work. You will need to issue the appropriate mdadm commands to fail/remove/add drives. I'd personally want to test that first on disks that do not contain important data. –  derobert Jul 19 '13 at 12:42

Is it possible to "lock" the RAID 1 Array to only work with specific disks/partitions independently from disk device names?

You should be able to use the aliases under /dev/disk/by-* to accomplish this. I would suggest /dev/disk/by-id/*, as those will directly map to information that should be readily visible on the physical disk, but any one of the directories should hold symlinks that will be useful in your scenario. Which ones to use is largely a matter of taste.

For example, on my system, /dev/disk/by-id/scsi-SATA_ST31000340NS_9QJ26FT9-part1 maps to /dev/sdb1. That is the bus, model, serial and partition number leading to the partition in question. If a drive in the array fails, you can use something like the mdadm command in your question to unambigiously determine which physical drive is the culprit.

Would it be, for example, sufficient to use the Array's UUID in /etc/fstab so as to be immune against changes of the disk device names?

I'm not really familiar with MD RAID, but would that not affect only the reference to the entire array? If you are worried about /dev/md0 suddenly showing up as /dev/md1, maybe there are other ways to provide a fixed reference to the array itself?

share|improve this answer
    
I have thought about /dev/disk/by-* but, being not an expert, I'd rather ask before I try. I don't think I worry about md0 becoming md-some-number, if this is possible at all. Rather, how not to loose the RAID set-up no matter if I plug-in a new internal or external disk that would show up with a device disk name already "taken" for some/all of the RAID-disks. –  Nikos Alexandris Jul 18 '13 at 12:36
    
Quotting: would that not affect only the reference to the entire array? Ehm, I guess so. So I am really after the disks that are parts of the RAID Array. –  Nikos Alexandris Jul 18 '13 at 12:39

Not the answer you're looking for? Browse other questions tagged or ask your own question.