2

I have a system with 10 SATA ports, and another SATA as the boot disk. The 10 SATA ports make up 5 software RAID1 arrays. The RAID disks may be removed between boots, and swapped in with arbitrary blank disks at any time.

I need to ensure that /dev/sda is always my first physical SATA port, and /dev/sdj is always the tenth for the RAID1 arrays to properly operate. If for example the first disk in the first port fails, that should be marked as a missing disk and so the disk in the next port should be /dev/sdb. Currently, the next available disk is mounted as /dev/sda - completely destroying my arrays - and my boot configuration.

Imagine a horrible scenario where every other disk fails, so each RAID1 array only has 1 working disk in it's pair. The numbering should be:

/dev/sda /dev/sdc /dev/sde /dev/sdg /dev/sdi

NOT:

/dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde

I have seen udev rules for labelling specific disks by UUIDs, but since users will be hotswapping disks arbitrarily this is not convenient at all.

By default Linux will label the next available disk with the next alphabetical character. There are many situations where a single broken disk will break multiple RAID 1 arrays.

  • How can I map a device to a specific hardware interface? Is this even possible?
  • Is it possible to have a "missing" device on boot, so subsequent devices do not get labelled incorrectly ?
  • Why do you need the drive letters and why would rearranging the letters destroy your archive? mdadm puts a unique UUID in the partitions that belong together. And that is what you should have in you mdadm.conf. What file is referring to these partitions by name? – Anthon Jan 16 '15 at 16:12
  • I may haved misunderstood how mdadm works but I am configuring my arrays using ARRAY /dev/md1 devices=/dev/sda1,/dev/sdb1 in mdadm.conf, but depending on which disks are available /dev/sda1 may correspond to a different disk than intended. I create the array with mdadm -Cv /dev/md1 -l1 -n2 /dev/sda1 /dev/sdb1. I would like to know how to assign /dev/sda to a particular SATA connection on my motherboard. – sg90 Jan 16 '15 at 18:37
1

If you create an array with:

mdadm --create --name=DATA00 --level=6 --auto=part --verbose /dev/md0 --raid-devices=6 /dev/sda1 /dev/sdb1 /dev/sdd1 /dev/sde1 /dev/sdf1 /dev/sdg1

and then do:

mdadm --detail --scan >> /etc/mdadm/mdadm.conf

you get an entry in mdadm.conf like:

ARRAY /dev/md/DATA00 metadata=1.2 name=owl:DATA00 UUID=5eeada67:ff994361:bae3ab52:d9e8bd49

There is no need to refer to the original partitions and/or driver ordering as the UUID takes care of this. Which actual partitions constitute the array after activation/reboot can be seen from /proc/mdstat. To look at an individual partition (including its UUID) use mdadm --examine /dev/sdXY

Given that there is never a need to have a particular order in your drives after reboot and since my BIOS switches things around depending on whether I have an external SATA attached or not, I am very happy that it doesn't matter.

  • I agree that in your example drive order does not matter, however I require multiple RAID devices in a system - I have /dev/md1 -> /dev/md4. The ordering does matter in that case, a disk intended for /dev/md1 could be incorrectly used in /dev/md2. – sg90 Jan 21 '15 at 0:13
  • I have found this on the Debian Wiki which seems to be a solution to this issue. I will post this as an answer if it resolves my issues on CentOS. – sg90 Jan 21 '15 at 1:05
  • @sg90 Why would the ordering matter? The partitions belonging to different RAID devices will have different UUIDs and that can solve which partition belongs to what drive. Why doesn't that solve the issue? – Anthon Jan 21 '15 at 6:49
1

The Debian Wiki has an excellent entry describing what I required. Following this I made my own rules under /etc/udev/rules.d/20-disk-bay.rules. I have only included the first two sata port mappings as an example:

# There are different DEVPATHs for major kernel versions!
# Example for SATA N:
#
# Kernel < 3 DEVPATH
# *1f.2/hostN/targetN:0:0/N:0:0:0*
#
# Kernel > 3 DEVPATH
# *1f.2/ata(N+1)/host*

########## Map SATA 0 to /dev/sdb ##############

# Kernel < 3

KERNEL=="sd?", SUBSYSTEM=="block", DEVPATH=="*1f.2/host0/target0:0:0/0:0:0:0*", NAME="sdb", RUN+="/usr/bin/logger My disk ATTR{partition}=$ATTR{partition}, DEVPATH=$devpath, ID_PATH=$ENV{ID_PATH}, ID_SERIAL=$ENV{ID_SERIAL}", GOTO="END_20_PERSISTENT_DISK"

KERNEL=="sd?*", ATTR{partition}=="1", SUBSYSTEM=="block", DEVPATH=="*1f.2/host0/target0:0:0/0:0:0:0*", NAME="sdb%n", RUN+="/usr/bin/logger My partition parent=%p number=%n, ATTR{partition}=$ATTR{partition}"

# Kernel > 3

KERNEL=="sd?", SUBSYSTEM=="block", DEVPATH=="*1f.2/ata1/host*", NAME="sdb", RUN+="/usr/bin/logger My disk ATTR{partition}=$ATTR{partition}, DEVPATH=$devpath, ID_PATH=$ENV{ID_PATH}, ID_SERIAL=$ENV{ID_SERIAL}", GOTO="END_20_PERSISTENT_DISK"

KERNEL=="sd?*", ATTR{partition}=="1", SUBSYSTEM=="block", DEVPATH=="*1f.2/ata1/host*", NAME="sdb%n", RUN+="/usr/bin/logger My partition parent=%p number=%n, ATTR{partition}=$ATTR{partition}"

########## Map SATA 1 to /dev/sdc ##############

# Kernel < 3

KERNEL=="sd?", SUBSYSTEM=="block", DEVPATH=="*1f.2/host1/target1:0:0/1:0:0:0*", NAME="sdc", RUN+="/usr/bin/logger My disk ATTR{partition}=$ATTR{partition}, DEVPATH=$devpath, ID_PATH=$ENV{ID_PATH}, ID_SERIAL=$ENV{ID_SERIAL}", GOTO="END_20_PERSISTENT_DISK"

KERNEL=="sd?*", ENV{DEVTYPE}=="partition", SUBSYSTEM=="block", DEVPATH=="*1f.2/host1/target1:0:0/1:0:0:0*", NAME="sdc%n", RUN+="/usr/bin/logger My partition parent=%p number=%n, ATTR{partition}=$ATTR{partition}"

# Kernel > 3

KERNEL=="sd?", SUBSYSTEM=="block", DEVPATH=="*1f.2/ata2/host*", NAME="sdc", RUN+="/usr/bin/logger My disk ATTR{partition}=$ATTR{partition}, DEVPATH=$devpath, ID_PATH=$ENV{ID_PATH}, ID_SERIAL=$ENV{ID_SERIAL}", GOTO="END_20_PERSISTENT_DISK"
KERNEL=="sd?*", ATTR{partition}=="1", SUBSYSTEM=="block", DEVPATH=="*1f.2/ata2/host*", NAME="sdc%n", RUN+="/usr/bin/logger My partition parent=%p number=%n, ATTR{partition}=$ATTR{partition}"

LABEL="END_20_PERSISTENT_DISK"

The rules above will always map any drive placed in SATA port 0, the first physical SATA port on my motherboard, as /dev/sdb and any drive placed in SATA 1 as /dev/sdc Consistent physical port mappings are critical in my use case, as I have 5 RAID-1 arrays where the disks can be arbitrarily swapped out of their physical hotswap bays. A non-technical user can swap out these disks at any time without having to deal with device IDs - the system is fully autonomous and will not construct the RAID arrays over the wrong disks in the hotswap bays. This is a very specific use case.

0

Why are users hot swapping disks arbitrarily?

Many hot swaps means the disks will most likely be worn faster. The raid advantage in speed will be lost. In fact, the performance of the array will probably be worse than a single disk while it's recovering.

The habit of ripping out disks arbitrarily will sooner or later land you in a situation where someone happens to pull out two disks belonging to the same array, likely crashing the system as a consequence.

Not to mention the problems you've described here. And I am not sure mdadm will pick up a new disk without it being manually added to the array, but there may be ways around that.

Is this a backup scheme? Then I highly discourage it! Use conventional backup - perhaps with LVM snapshots if you have LVM.

If users are "abusing" your servers, put a lock on the server room door... or deprive them of the keys supplied with most hot swap bays... or inform management that the system - important enough to have raid - is risking down time due to this behavior.

Do not hot swap disks unless the disks are broken or to be replaced due to old age!

Hot swapping is a tool to allow uninterrupted system uptime when replacing disks that are broken, old or needs to be replaced for some other, irregular reason - that is pretty much all it's good for (which is good enough when it comes to very important systems).

When a disk breaks you can either have an extra spare disk added to the raid array beforehand, causing the spare disk to be synced automatically, or you will have to add the hot swapped replacement disk manually.

See Anthon's answer, or this guide: https://raid.wiki.kernel.org/index.php/RAID_setup

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.