1

I am using a SuperMicro X9SRi-3F motherboard, which uses the Intel C606 chipset and has a dual 4-port SCU with SAS capability. The HDDs I'm using are 4x WD Re's (WD6001F9YZ), and 2x WD Gold's (WD6002FRYZ). All are 6TB in capacity.

The WD6001F9YZ's are connected to ports 0-3, and the WD6002FRYZ's are on ports 4&5 (should be noted, I will eventually be adding two more WD6001F9YZ's to ports 6&7).

OS is installed on a 1TB drive connected to one of the normal SATA 2.0 ports on the board.

CentOS 7 can recognize all the drives, as lsblk outputs the following:

NAME            MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda               8:0    0   5.5T  0 disk 
sdb               8:16   0   5.5T  0 disk 
sdc               8:32   0   5.5T  0 disk 
sdd               8:48   0   5.5T  0 disk 
sde               8:64   0   5.5T  0 disk 
sdf               8:80   0   5.5T  0 disk 

They also all show up under /dev/disk/by-id:

lrwxrwxrwx. 1 root root  9 Oct 17 14:06 ata-WDC_WD6001F9YZ-09YUWL1_WD-WX41DA58PYRS -> ../../sda
lrwxrwxrwx. 1 root root  9 Oct 17 14:06 ata-WDC_WD6001F9YZ-09YUWL1_WD-WX41DA5LV7KR -> ../../sdb
lrwxrwxrwx. 1 root root  9 Oct 17 14:06 ata-WDC_WD6001F9YZ-09YUWL1_WD-WX41DA5LV87E -> ../../sdc
lrwxrwxrwx. 1 root root  9 Oct 17 14:06 ata-WDC_WD6001F9YZ-09YUWL1_WD-WX41DA5LVL4J -> ../../sdd
lrwxrwxrwx. 1 root root  9 Oct 17 14:06 ata-WDC_WD6002FRYZ-01WD5B0_K1HNL2KD -> ../../sde
lrwxrwxrwx. 1 root root  9 Oct 17 14:06 ata-WDC_WD6002FRYZ-01WD5B0_K1JMY3ND -> ../../sdf

However, only drives c-f show up in /dev/disk/by-path:

lrwxrwxrwx. 1 root root  9 Oct 17 13:42 pci-0000:03:00.0-sas-0x500304801349fe00-lun-0 -> ../../sde
lrwxrwxrwx. 1 root root  9 Oct 17 13:42 pci-0000:03:00.0-sas-0x500304801349fe01-lun-0 -> ../../sdf
lrwxrwxrwx. 1 root root  9 Oct 17 13:42 pci-0000:03:00.0-sas-0x500304801349fe02-lun-0 -> ../../sdc
lrwxrwxrwx. 1 root root  9 Oct 17 13:42 pci-0000:03:00.0-sas-0x500304801349fe03-lun-0 -> ../../sdd
lrwxrwxrwx. 1 root root  9 Oct 17 13:42 pci-0000:03:00.0-sas-phy0-lun-0 -> ../../sde
lrwxrwxrwx. 1 root root  9 Oct 17 13:42 pci-0000:03:00.0-sas-phy1-lun-0 -> ../../sdf
lrwxrwxrwx. 1 root root  9 Oct 17 13:42 pci-0000:03:00.0-sas-phy2-lun-0 -> ../../sdc
lrwxrwxrwx. 1 root root  9 Oct 17 13:42 pci-0000:03:00.0-sas-phy3-lun-0 -> ../../sdd

In the MB's BIOS, when the SCU unit is enabled and the "SCU RAID Option ROM/UEFI Driver" is enabled, all of the drives are recognized:

Option ROM Enabled

When the SCU unit is enabled, but the Option ROM is disabled, the drives do not show up in the BIOS:

Option ROM Disabled

Both settings (Option ROM Disabled/Enabled) result in the same issue. Of course, disabling the SCU means it doesn't show up in the OS at all (disappears from the output of lspci)

The output of lshw -c storage looks like this:

*-sas                     
   description: Serial Attached SCSI controller
   product: C606 chipset Dual 4-Port SATA/SAS Storage Control Unit
   vendor: Intel Corporation
   physical id: 0
   bus info: pci@0000:03:00.0
   logical name: scsi6
   logical name: scsi7
   version: 06
   width: 64 bits
   clock: 33MHz
   capabilities: sas pm pciexpress msix bus_master cap_list
   configuration: driver=isci latency=0
   resources: irq:49 memory:fa8f8000-fa8fffff memory:fa000000-fa7fffff ioport:e100(size=256) ioport:e000(size=256) memory:fa800000-fa8f7fff

This is an issue for me as I intend to set up an 8-disk ZFS array, and want to reference the drive's physical location in the system (slot1, slot2, etc).

My intuition tells me that it's an issue with the kernel driver for the C606, but I honestly have no idea.

[EDIT]

If I hotplug the other SATA drives I intend on using on ports 6&7, /dev/disk/by-path looks like this:

lrwxrwxrwx. 1 root root  9 Oct 17 14:20 pci-0000:03:00.0-sas-0x5fcfffff00000001-lun-0 -> ../../sda
lrwxrwxrwx. 1 root root  9 Oct 17 14:20 pci-0000:03:00.0-sas-0x5fcfffff00000002-lun-0 -> ../../sde
lrwxrwxrwx. 1 root root  9 Oct 17 14:20 pci-0000:03:00.0-sas-0x5fcfffff00000003-lun-0 -> ../../sdc
lrwxrwxrwx. 1 root root  9 Oct 17 14:24 pci-0000:03:00.0-sas-0x5fcfffff00000004-lun-0 -> ../../sdi
lrwxrwxrwx. 1 root root 10 Oct 17 14:24 pci-0000:03:00.0-sas-0x5fcfffff00000004-lun-0-part1 -> ../../sdi1
lrwxrwxrwx. 1 root root 10 Oct 17 14:24 pci-0000:03:00.0-sas-0x5fcfffff00000004-lun-0-part2 -> ../../sdi2
lrwxrwxrwx. 1 root root  9 Oct 17 14:24 pci-0000:03:00.0-sas-0x5fcfffff00000005-lun-0 -> ../../sdj
lrwxrwxrwx. 1 root root 10 Oct 17 14:24 pci-0000:03:00.0-sas-0x5fcfffff00000005-lun-0-part1 -> ../../sdj1
lrwxrwxrwx. 1 root root 10 Oct 17 14:24 pci-0000:03:00.0-sas-0x5fcfffff00000005-lun-0-part2 -> ../../sdj2
lrwxrwxrwx. 1 root root  9 Oct 17 14:20 pci-0000:03:00.0-sas-phy0-lun-0 -> ../../sde
lrwxrwxrwx. 1 root root  9 Oct 17 14:20 pci-0000:03:00.0-sas-phy1-lun-0 -> ../../sdb
lrwxrwxrwx. 1 root root  9 Oct 17 14:24 pci-0000:03:00.0-sas-phy2-lun-0 -> ../../sdi
lrwxrwxrwx. 1 root root 10 Oct 17 14:24 pci-0000:03:00.0-sas-phy2-lun-0-part1 -> ../../sdi1
lrwxrwxrwx. 1 root root 10 Oct 17 14:24 pci-0000:03:00.0-sas-phy2-lun-0-part2 -> ../../sdi2
lrwxrwxrwx. 1 root root  9 Oct 17 14:24 pci-0000:03:00.0-sas-phy3-lun-0 -> ../../sdj
lrwxrwxrwx. 1 root root 10 Oct 17 14:24 pci-0000:03:00.0-sas-phy3-lun-0-part1 -> ../../sdj1
lrwxrwxrwx. 1 root root 10 Oct 17 14:24 pci-0000:03:00.0-sas-phy3-lun-0-part2 -> ../../sdj2

The portion of the path that previously started with sas-0x500304801349fe## has changed to sas-0x5fcfffff000000## because I've disabled the Option ROM in the BIOS. But what's interesting here is that sdd and sdf have disappeared from the by-path directory, but sda, sdi, and sdj have appeared. In addition, the order has changed.

Of course, all eight drives show up in the by-id directory:

lrwxrwxrwx. 1 root root  9 Oct 17 14:20 ata-WDC_WD6001F9YZ-09YUWL1_WD-WX41DA58PYRS -> ../../sda
lrwxrwxrwx. 1 root root  9 Oct 17 14:20 ata-WDC_WD6001F9YZ-09YUWL1_WD-WX41DA5LV7KR -> ../../sdb
lrwxrwxrwx. 1 root root  9 Oct 17 14:20 ata-WDC_WD6001F9YZ-09YUWL1_WD-WX41DA5LV87E -> ../../sdc
lrwxrwxrwx. 1 root root  9 Oct 17 14:24 ata-WDC_WD6001F9YZ-09YUWL1_WD-WX41DA5LVJK2 -> ../../sdj
lrwxrwxrwx. 1 root root 10 Oct 17 14:24 ata-WDC_WD6001F9YZ-09YUWL1_WD-WX41DA5LVJK2-part1 -> ../../sdj1
lrwxrwxrwx. 1 root root 10 Oct 17 14:24 ata-WDC_WD6001F9YZ-09YUWL1_WD-WX41DA5LVJK2-part2 -> ../../sdj2
lrwxrwxrwx. 1 root root  9 Oct 17 14:20 ata-WDC_WD6001F9YZ-09YUWL1_WD-WX41DA5LVL4J -> ../../sdd
lrwxrwxrwx. 1 root root  9 Oct 17 14:24 ata-WDC_WD6001F9YZ-09YUWL1_WD-WX41DA5LVNXX -> ../../sdi
lrwxrwxrwx. 1 root root 10 Oct 17 14:24 ata-WDC_WD6001F9YZ-09YUWL1_WD-WX41DA5LVNXX-part1 -> ../../sdi1
lrwxrwxrwx. 1 root root 10 Oct 17 14:24 ata-WDC_WD6001F9YZ-09YUWL1_WD-WX41DA5LVNXX-part2 -> ../../sdi2
lrwxrwxrwx. 1 root root  9 Oct 17 14:20 ata-WDC_WD6002FRYZ-01WD5B0_K1HNL2KD -> ../../sde
lrwxrwxrwx. 1 root root  9 Oct 17 14:20 ata-WDC_WD6002FRYZ-01WD5B0_K1JMY3ND -> ../../sdf

[EDIT 2]

The by-path directory seems to change with every boot now. It currently looks like this:

lrwxrwxrwx. 1 root root  9 Oct 17 15:03 pci-0000:03:00.0-sas-0x5fcfffff00000001-lun-0 -> ../../sda
lrwxrwxrwx. 1 root root  9 Oct 17 15:03 pci-0000:03:00.0-sas-0x5fcfffff00000002-lun-0 -> ../../sdb
lrwxrwxrwx. 1 root root  9 Oct 17 15:03 pci-0000:03:00.0-sas-0x5fcfffff00000003-lun-0 -> ../../sdf
lrwxrwxrwx. 1 root root  9 Oct 17 15:03 pci-0000:03:00.0-sas-0x5fcfffff00000004-lun-0 -> ../../sdd
lrwxrwxrwx. 1 root root  9 Oct 17 15:03 pci-0000:03:00.0-sas-phy0-lun-0 -> ../../sda
lrwxrwxrwx. 1 root root  9 Oct 17 15:03 pci-0000:03:00.0-sas-phy1-lun-0 -> ../../sdf
lrwxrwxrwx. 1 root root  9 Oct 17 15:03 pci-0000:03:00.0-sas-phy2-lun-0 -> ../../sdc
lrwxrwxrwx. 1 root root  9 Oct 17 15:03 pci-0000:03:00.0-sas-phy3-lun-0 -> ../../sdd

While the by-id path looks like this:

lrwxrwxrwx. 1 root root  9 Oct 17 15:03 ata-WDC_WD6001F9YZ-09YUWL1_WD-WX41DA58PYRS -> ../../sda
lrwxrwxrwx. 1 root root  9 Oct 17 15:03 ata-WDC_WD6001F9YZ-09YUWL1_WD-WX41DA5LV7KR -> ../../sdb
lrwxrwxrwx. 1 root root  9 Oct 17 15:03 ata-WDC_WD6001F9YZ-09YUWL1_WD-WX41DA5LV87E -> ../../sdc
lrwxrwxrwx. 1 root root  9 Oct 17 15:03 ata-WDC_WD6001F9YZ-09YUWL1_WD-WX41DA5LVL4J -> ../../sdd
lrwxrwxrwx. 1 root root  9 Oct 17 15:03 ata-WDC_WD6002FRYZ-01WD5B0_K1HNL2KD -> ../../sde
lrwxrwxrwx. 1 root root  9 Oct 17 15:03 ata-WDC_WD6002FRYZ-01WD5B0_K1JMY3ND -> ../../sdf

The disks are consistently assigned the same device ID (sda/sdb/etc), which is reassuring, but the path changes unpredictably (making it completely unusable for my purposes).

[EDIT 3]

The output of du -a /sys/devices/pci0000\:00/0000:00:01.0/0000:01:00.0/0000:02:08.0/0000:03:00.0 | grep -E 'sd.' | grep -vE 'sd./' shows the correct mapping for the drives:

0   /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:08.0/0000:03:00.0/host6/port-6:0/end_device-6:0/target6:0:0/6:0:0:0/block/sda
0   /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:08.0/0000:03:00.0/host6/port-6:1/end_device-6:1/target6:0:1/6:0:1:0/block/sdb
0   /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:08.0/0000:03:00.0/host6/port-6:2/end_device-6:2/target6:0:2/6:0:2:0/block/sdc
0   /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:08.0/0000:03:00.0/host6/port-6:3/end_device-6:3/target6:0:3/6:0:3:0/block/sdd
0   /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:08.0/0000:03:00.0/host7/port-7:0/end_device-7:0/target7:0:0/7:0:0:0/block/sde
0   /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:08.0/0000:03:00.0/host7/port-7:1/end_device-7:1/target7:0:1/7:0:1:0/block/sdf

So I think this issue is related to a bug in udev

1

I also have a Dell T420 server, and the output of udevadm info gives me this:

E: DEVPATH=/devices/pci0000:00/0000:00:01.0/0000:08:00.0/host0/target0:0:0/0:0:0:0/block/sda
E: DEVPATH=/devices/pci0000:00/0000:00:01.0/0000:08:00.0/host0/target0:0:1/0:0:1:0/block/sdb
E: DEVPATH=/devices/pci0000:00/0000:00:01.0/0000:08:00.0/host0/target0:0:2/0:0:2:0/block/sdc
E: DEVPATH=/devices/pci0000:00/0000:00:01.0/0000:08:00.0/host0/target0:0:3/0:0:3:0/block/sdd
E: DEVPATH=/devices/pci0000:00/0000:00:01.0/0000:08:00.0/host0/target0:0:4/0:0:4:0/block/sde
E: DEVPATH=/devices/pci0000:00/0000:00:01.0/0000:08:00.0/host0/target0:0:5/0:0:5:0/block/sdf
E: DEVPATH=/devices/pci0000:00/0000:00:01.0/0000:08:00.0/host0/target0:0:6/0:0:6:0/block/sdg
E: DEVPATH=/devices/pci0000:00/0000:00:01.0/0000:08:00.0/host0/target0:0:7/0:0:7:0/block/sdh

As you can see, there is only on instance of host# in the device path (host0) whereas the intel SCU has two (host6 and host7). Apparently udev on CentOS 7 doesn't know how to handle this properly, and simply overwrites the device links (so any device under the host6 node gets its symlink in /dev/disks/by-path overwritten by the corresponding device under the host7 node).

Looks like I need learn how to write udev rules now....

[EDIT]

Initial attempt at a udev rule to solve the problem: https://gist.github.com/Toasty27/49da6175371cdde317e662fb8a7d078a

It's very ugly and buggy. Doesn't handle hot-plugging properly at all, and doesn't do anything with partitions. I need to find a way to create the updated properties from existing info instead of editing them in-place, otherwise there's a chance of the paths changing every time the udev rules are reloaded.

[EDIT 2]

Gist has been updated to output paths similar to what's produced by the handle_scsi_default function in the path_id builtin. It's much, much more reliable now, and also handles partitions. Hopefully someone else will also find it useful. It's still a hack though, so YMMV.

Currently working to get a proper fix patched into udev.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.