How to get Linux to completely ignore a SCSI drive?

Question

Similar to this question I a am interested in completely ignoring a drive, but in my case it is one drive which is exposed to the system as a SCSI drive. I have two drives from 21 drives in the server failing and failing:

[2524080.689492] scsi 0:0:90900:0: Direct-Access     ATA      ST3000DM001-1CH1 CC43 PQ: 0 ANSI: 6
[2524080.689502] scsi 0:0:90900:0: SATA: handle(0x000d), sas_addr(0x5003048001f298cf), phy(15), device_name(0x0000000000000000)
[2524080.689506] scsi 0:0:90900:0: SATA: enclosure_logical_id(0x5003048001f298ff), slot(3)
[2524080.689594] scsi 0:0:90900:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
[2524080.690671] sd 0:0:90900:0: tag#1 CDB: Test Unit Ready 00 00 00 00 00 00
[2524080.690680] mpt2sas_cm0:   sas_address(0x5003048001f298cf), phy(15)
[2524080.690683] mpt2sas_cm0:   enclosure_logical_id(0x5003048001f298ff),slot(3)
[2524080.690686] mpt2sas_cm0:   handle(0x000d), ioc_status(success)(0x0000), smid(17)
[2524080.690695] mpt2sas_cm0:   request_len(0), underflow(0), resid(0)
[2524080.690698] mpt2sas_cm0:   tag(65535), transfer_count(0), sc->result(0x00000000)
[2524080.690701] mpt2sas_cm0:   scsi_status(check condition)(0x02), scsi_state(autosense valid )(0x01)
[2524080.690704] mpt2sas_cm0:   [sense_key,asc,ascq]: [0x06,0x29,0x00], count(18)
[2524080.690728] sd 0:0:90900:0: Attached scsi generic sg0 type 0
[2524080.691269] sd 0:0:90900:0: [sdb] 5860533168 512-byte logical blocks: (3.00 TB/2.73 TiB)
[2524080.691285] sd 0:0:90900:0: [sdb] 4096-byte physical blocks
[2524111.163712] sd 0:0:90900:0: attempting task abort! scmd(ffff880869121800)
[2524111.163722] sd 0:0:90900:0: tag#2 CDB: Mode Sense(6) 1a 00 3f 00 04 00
[2524111.163729] scsi target0:0:90900: handle(0x000d), sas_address(0x5003048001f298cf), phy(15)
[2524111.163733] scsi target0:0:90900: enclosure_logical_id(0x5003048001f298ff), slot(3)
[2524111.442310] sd 0:0:90900:0: device_block, handle(0x000d)
[2524113.442331] sd 0:0:90900:0: device_unblock and setting to running, handle(0x000d)
[2524114.939280] sd 0:0:90900:0: task abort: SUCCESS scmd(ffff880869121800)
[2524114.939358] sd 0:0:90900:0: [sdb] Write Protect is off
[2524114.939366] sd 0:0:90900:0: [sdb] Mode Sense: 00 00 00 00
[2524114.939444] sd 0:0:90900:0: [sdb] Asking for cache data failed
[2524114.939501] sd 0:0:90900:0: [sdb] Assuming drive cache: write through
[2524114.940380] sd 0:0:90900:0: [sdb] Read Capacity(16) failed: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
[2524114.940387] sd 0:0:90900:0: [sdb] Sense not available.
[2524114.940566] sd 0:0:90900:0: [sdb] Read Capacity(10) failed: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
[2524114.940570] sd 0:0:90900:0: [sdb] Sense not available.
[2524114.940778] sd 0:0:90900:0: [sdb] Attached SCSI disk
[2524114.984489] mpt2sas_cm0: removing handle(0x000d), sas_addr(0x5003048001f298cf)
[2524114.984494] mpt2sas_cm0: removing : enclosure logical id(0x5003048001f298ff), slot(3)
[2524134.939383] mpt2sas_cm0: log_info(0x31111000): originator(PL), code(0x11), sub_code(0x1000)
[2524134.940116] mpt2sas_cm0: removing handle(0x000e), sas_addr(0x5003048001f298d0)
[2524134.940122] mpt2sas_cm0: removing enclosure logical id(0x5003048001f298ff), slot(4)
[2524153.940404] scsi 0:0:90902:0: Direct-Access     ATA      ST3000DM001-1CH1 CC43 PQ: 0 ANSI: 6
[2524153.940418] scsi 0:0:90902:0: SATA: handle(0x000d), sas_addr(0x5003048001f298cf), phy(15), device_name(0x0000000000000000)
[2524153.940423] scsi 0:0:90902:0: SATA: enclosure_logical_id(0x5003048001f298ff), slot(3)
[2524153.940699] scsi 0:0:90902:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
[2524153.942194] sd 0:0:90902:0: tag#0 CDB: Test Unit Ready 00 00 00 00 00 00
[2524153.942205] mpt2sas_cm0:   sas_address(0x5003048001f298cf), phy(15)
[2524153.942208] mpt2sas_cm0:   enclosure_logical_id(0x5003048001f298ff),slot(3)
[2524153.942212] mpt2sas_cm0:   handle(0x000d), ioc_status(success)(0x0000), smid(12)
[2524153.942214] mpt2sas_cm0:   request_len(0), underflow(0), resid(0)
[2524153.942217] mpt2sas_cm0:   tag(65535), transfer_count(0), sc->result(0x00000000)
[2524153.942220] mpt2sas_cm0:   scsi_status(check condition)(0x02), scsi_state(autosense valid )(0x01)
[2524153.942223] mpt2sas_cm0:   [sense_key,asc,ascq]: [0x06,0x29,0x00], count(18)
[2524153.942361] sd 0:0:90902:0: Attached scsi generic sg0 type 0
[2524153.942833] sd 0:0:90902:0: [sdb] 5860533168 512-byte logical blocks: (3.00 TB/2.73 TiB)
[2524153.942840] sd 0:0:90902:0: [sdb] 4096-byte physical blocks
[2524154.190159] scsi 0:0:90903:0: Direct-Access     ATA      ST3000DM001-1CH1 CC43 PQ: 0 ANSI: 6
[2524154.190174] scsi 0:0:90903:0: SATA: handle(0x0022), sas_addr(0x5003048001ec55ed), phy(13), device_name(0x0000000000000000)
[2524154.190179] scsi 0:0:90903:0: SATA: enclosure_logical_id(0x5003048001ec55ff), slot(1)
[2524154.190368] scsi 0:0:90903:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
[2524154.191634] sd 0:0:90903:0: tag#1 CDB: Test Unit Ready 00 00 00 00 00 00
[2524154.191639] mpt2sas_cm0:   sas_address(0x5003048001ec55ed), phy(13)
[2524154.191642] mpt2sas_cm0:   enclosure_logical_id(0x5003048001ec55ff),slot(1)
[2524154.191645] mpt2sas_cm0:   handle(0x0022), ioc_status(success)(0x0000), smid(12)
[2524154.191648] mpt2sas_cm0:   request_len(0), underflow(0), resid(0)
[2524154.191651] mpt2sas_cm0:   tag(65535), transfer_count(0), sc->result(0x00000000)
[2524154.191654] mpt2sas_cm0:   scsi_status(check condition)(0x02), scsi_state(autosense valid )(0x01)
[2524154.191657] mpt2sas_cm0:   [sense_key,asc,ascq]: [0x06,0x29,0x00], count(18)
[2524154.191800] sd 0:0:90903:0: Attached scsi generic sg3 type 0
[2524154.192211] sd 0:0:90903:0: [sdd] 5860533168 512-byte logical blocks: (3.00 TB/2.73 TiB)
[2524154.192219] sd 0:0:90903:0: [sdd] 4096-byte physical blocks

This is in our case an old server we have decided not to upgrade/fix. And I am now thinking about even not removing old drives out, just leaving them in, making array smaller, and disabling them. The array is not full, and we are using it only as an additional backup location for some other servers.

So, me being lazy and not wanting to go to a server room, is there a way to just disable those drives and move on? :-)

More information about the system:

lspci -nn -v -s 05:00.0:

05:00.0 Serial Attached SCSI controller [0107]: LSI Logic / Symbios Logic SAS2308 PCI-Express Fusion-MPT SAS-2 [1000:0087] (rev 05)
    Subsystem: LSI Logic / Symbios Logic SAS2308 PCI-Express Fusion-MPT SAS-2 [1000:3020]
    Flags: bus master, fast devsel, latency 0, IRQ 29
    I/O ports at 7000 [size=256]
    Memory at df640000 (64-bit, non-prefetchable) [size=64K]
    Memory at df600000 (64-bit, non-prefetchable) [size=256K]
    Expansion ROM at df500000 [disabled] [size=1M]
    Capabilities: [50] Power Management version 3
    Capabilities: [68] Express Endpoint, MSI 00
    Capabilities: [d0] Vital Product Data
    Capabilities: [a8] MSI: Enable- Count=1/1 Maskable- 64bit+
    Capabilities: [c0] MSI-X: Enable+ Count=16 Masked-
    Capabilities: [100] Advanced Error Reporting
    Capabilities: [1e0] #19
    Capabilities: [1c0] Power Budgeting <?>
    Capabilities: [190] #16
    Capabilities: [148] Alternative Routing-ID Interpretation (ARI)
    Kernel driver in use: mpt3sas
    Kernel modules: mpt3sas

lsscsi -v:

[0:0:3:0]    disk    ATA      ST3000DM001-1CH1 CC43  /dev/sdc 
  dir: /sys/bus/scsi/devices/0:0:3:0  [/sys/devices/pci0000:00/0000:00:03.0/0000:05:00.0/host0/port-0:0/expander-0:0/port-0:0:2/end_device-0:0:2/target0:0:3/0:0:3:0]
[0:0:6:0]    disk    ATA      ST3000DM001-1CH1 CC43  /dev/sdf 
  dir: /sys/bus/scsi/devices/0:0:6:0  [/sys/devices/pci0000:00/0000:00:03.0/0000:05:00.0/host0/port-0:0/expander-0:0/port-0:0:5/end_device-0:0:5/target0:0:6/0:0:6:0]
[0:0:7:0]    disk    ATA      ST3000DM001-1CH1 CC43  /dev/sdg 
  dir: /sys/bus/scsi/devices/0:0:7:0  [/sys/devices/pci0000:00/0000:00:03.0/0000:05:00.0/host0/port-0:0/expander-0:0/port-0:0:6/end_device-0:0:6/target0:0:7/0:0:7:0]
[0:0:8:0]    disk    ATA      ST3000DM001-1CH1 CC43  /dev/sdh 
  dir: /sys/bus/scsi/devices/0:0:8:0  [/sys/devices/pci0000:00/0000:00:03.0/0000:05:00.0/host0/port-0:0/expander-0:0/port-0:0:7/end_device-0:0:7/target0:0:8/0:0:8:0]
[0:0:11:0]   disk    ATA      ST3000DM001-1CH1 CC43  /dev/sdi 
  dir: /sys/bus/scsi/devices/0:0:11:0  [/sys/devices/pci0000:00/0000:00:03.0/0000:05:00.0/host0/port-0:0/expander-0:0/port-0:0:10/end_device-0:0:10/target0:0:11/0:0:11:0]
[0:0:12:0]   disk    ATA      ST3000DM001-1CH1 CC43  /dev/sdj 
  dir: /sys/bus/scsi/devices/0:0:12:0  [/sys/devices/pci0000:00/0000:00:03.0/0000:05:00.0/host0/port-0:0/expander-0:0/port-0:0:11/end_device-0:0:11/target0:0:12/0:0:12:0]
[0:0:13:0]   disk    ATA      ST3000DM001-1CH1 CC43  /dev/sdk 
  dir: /sys/bus/scsi/devices/0:0:13:0  [/sys/devices/pci0000:00/0000:00:03.0/0000:05:00.0/host0/port-0:0/expander-0:0/port-0:0:12/end_device-0:0:12/target0:0:13/0:0:13:0]
[0:0:15:0]   disk    ATA      ST3000DM001-1CH1 CC43  /dev/sdl 
  dir: /sys/bus/scsi/devices/0:0:15:0  [/sys/devices/pci0000:00/0000:00:03.0/0000:05:00.0/host0/port-0:0/expander-0:0/port-0:0:14/end_device-0:0:14/target0:0:15/0:0:15:0]
[0:0:16:0]   disk    ATA      ST3000DM001-1CH1 CC43  /dev/sdm 
  dir: /sys/bus/scsi/devices/0:0:16:0  [/sys/devices/pci0000:00/0000:00:03.0/0000:05:00.0/host0/port-0:0/expander-0:0/port-0:0:15/end_device-0:0:15/target0:0:16/0:0:16:0]
[0:0:18:0]   disk    ATA      ST3000DM001-1CH1 CC43  /dev/sdn 
  dir: /sys/bus/scsi/devices/0:0:18:0  [/sys/devices/pci0000:00/0000:00:03.0/0000:05:00.0/host0/port-0:0/expander-0:0/port-0:0:17/end_device-0:0:17/target0:0:18/0:0:18:0]
[0:0:20:0]   disk    ATA      ST3000DM001-1CH1 CC43  /dev/sdo 
  dir: /sys/bus/scsi/devices/0:0:20:0  [/sys/devices/pci0000:00/0000:00:03.0/0000:05:00.0/host0/port-0:0/expander-0:0/port-0:0:19/end_device-0:0:19/target0:0:20/0:0:20:0]
[0:0:21:0]   disk    ATA      ST3000DM001-1CH1 CC43  /dev/sdp 
  dir: /sys/bus/scsi/devices/0:0:21:0  [/sys/devices/pci0000:00/0000:00:03.0/0000:05:00.0/host0/port-0:0/expander-0:0/port-0:0:20/end_device-0:0:20/target0:0:21/0:0:21:0]
[0:0:22:0]   enclosu LSI CORP SAS2X36          0717  -        
  dir: /sys/bus/scsi/devices/0:0:22:0  [/sys/devices/pci0000:00/0000:00:03.0/0000:05:00.0/host0/port-0:0/expander-0:0/port-0:0:21/end_device-0:0:21/target0:0:22/0:0:22:0]
[0:0:23:0]   disk    ATA      ST3000DM001-1CH1 CC43  /dev/sdq 
  dir: /sys/bus/scsi/devices/0:0:23:0  [/sys/devices/pci0000:00/0000:00:03.0/0000:05:00.0/host0/port-0:1/expander-0:1/port-0:1:1/end_device-0:1:1/target0:0:23/0:0:23:0]
[0:0:24:0]   disk    ATA      ST3000DM001-1CH1 CC43  /dev/sdr 
  dir: /sys/bus/scsi/devices/0:0:24:0  [/sys/devices/pci0000:00/0000:00:03.0/0000:05:00.0/host0/port-0:1/expander-0:1/port-0:1:2/end_device-0:1:2/target0:0:24/0:0:24:0]
[0:0:25:0]   disk    ATA      ST3000DM001-1CH1 CC43  /dev/sds 
  dir: /sys/bus/scsi/devices/0:0:25:0  [/sys/devices/pci0000:00/0000:00:03.0/0000:05:00.0/host0/port-0:1/expander-0:1/port-0:1:3/end_device-0:1:3/target0:0:25/0:0:25:0]
[0:0:26:0]   disk    ATA      ST3000DM001-1CH1 CC43  /dev/sdt 
  dir: /sys/bus/scsi/devices/0:0:26:0  [/sys/devices/pci0000:00/0000:00:03.0/0000:05:00.0/host0/port-0:1/expander-0:1/port-0:1:4/end_device-0:1:4/target0:0:26/0:0:26:0]
[0:0:28:0]   disk    ATA      ST3000DM001-1CH1 CC43  /dev/sdu 
  dir: /sys/bus/scsi/devices/0:0:28:0  [/sys/devices/pci0000:00/0000:00:03.0/0000:05:00.0/host0/port-0:1/expander-0:1/port-0:1:6/end_device-0:1:6/target0:0:28/0:0:28:0]
[0:0:30:0]   disk    ATA      ST3000DM001-1CH1 CC43  /dev/sdw 
  dir: /sys/bus/scsi/devices/0:0:30:0  [/sys/devices/pci0000:00/0000:00:03.0/0000:05:00.0/host0/port-0:1/expander-0:1/port-0:1:8/end_device-0:1:8/target0:0:30/0:0:30:0]
[0:0:31:0]   disk    ATA      ST3000DM001-1CH1 CC43  /dev/sdx 
  dir: /sys/bus/scsi/devices/0:0:31:0  [/sys/devices/pci0000:00/0000:00:03.0/0000:05:00.0/host0/port-0:1/expander-0:1/port-0:1:9/end_device-0:1:9/target0:0:31/0:0:31:0]
[0:0:34:0]   enclosu LSI CORP SAS2X28          0717  -        
  dir: /sys/bus/scsi/devices/0:0:34:0  [/sys/devices/pci0000:00/0000:00:03.0/0000:05:00.0/host0/port-0:1/expander-0:1/port-0:1:12/end_device-0:1:12/target0:0:34/0:0:34:0]
[0:0:25856:0]disk    ATA      ST3000DM001-1CH1 CC43  /dev/sda 
  dir: /sys/bus/scsi/devices/0:0:25856:0  [/sys/devices/pci0000:00/0000:00:03.0/0000:05:00.0/host0/port-0:0/expander-0:0/port-0:0:14357/end_device-0:0:14357/target0:0:25856/0:0:25856:0]
[0:0:98760:0]disk    ATA      ST3000DM001-1CH1 CC43  -        
  dir: /sys/bus/scsi/devices/0:0:98760:0  [/sys/devices/pci0000:00/0000:00:03.0/0000:05:00.0/host0/port-0:0/expander-0:0/port-0:0:60931/end_device-0:0:60931/target0:0:98760/0:0:98760:0]
[2:0:0:0]    disk    ATA      PLEXTOR PX-128M5 1.00  /dev/sdy 
  dir: /sys/bus/scsi/devices/2:0:0:0  [/sys/devices/pci0000:00/0000:00:1f.2/ata2/host2/target2:0:0/2:0:0:0]

lsscsi -Hv:

[0]    mpt2sas       
  dir: /sys/class/scsi_host//host0
  device dir: /sys/devices/pci0000:00/0000:00:03.0/0000:05:00.0/host0
[1]    ahci          
  dir: /sys/class/scsi_host//host1
  device dir: /sys/devices/pci0000:00/0000:00:1f.2/ata1/host1
[2]    ahci          
  dir: /sys/class/scsi_host//host2
  device dir: /sys/devices/pci0000:00/0000:00:1f.2/ata2/host2
[3]    ahci          
  dir: /sys/class/scsi_host//host3
  device dir: /sys/devices/pci0000:00/0000:00:1f.2/ata3/host3
[4]    ahci          
  dir: /sys/class/scsi_host//host4
  device dir: /sys/devices/pci0000:00/0000:00:1f.2/ata4/host4
[5]    ahci          
  dir: /sys/class/scsi_host//host5
  device dir: /sys/devices/pci0000:00/0000:00:1f.2/ata5/host5
[6]    ahci          
  dir: /sys/class/scsi_host//host6
  device dir: /sys/devices/pci0000:00/0000:00:1f.2/ata6/host6

smp_discover /dev/bsg/expander-0:0:

  phy   0:S:attached:[500605b00507dd20:03  i(SSP+STP+SMP)]  6 Gbps
  phy   1:S:attached:[500605b00507dd20:02  i(SSP+STP+SMP)]  6 Gbps
  phy   2:S:attached:[500605b00507dd20:01  i(SSP+STP+SMP)]  6 Gbps
  phy   3:S:attached:[500605b00507dd20:00  i(SSP+STP+SMP)]  6 Gbps
  phy  12:U:attached:[5003048001f298cc:00  t(SATA)]  6 Gbps
  phy  13:U:attached:[5003048001f298cd:00  t(SATA)]  6 Gbps
  phy  14:U:attached:[5003048001f298ce:00  t(SATA)]  6 Gbps
  phy  17:U:attached:[5003048001f298d1:00  t(SATA)]  6 Gbps
  phy  19:U:attached:[5003048001f298d3:00  t(SATA)]  6 Gbps
  phy  20:U:attached:[5003048001f298d4:00  t(SATA)]  6 Gbps
  phy  21:U:attached:[5003048001f298d5:00  t(SATA)]  6 Gbps
  phy  22:U:attached:[5003048001f298d6:00  t(SATA)]  6 Gbps
  phy  23:U:attached:[5003048001f298d7:00  t(SATA)]  6 Gbps
  phy  25:U:attached:[5003048001f298d9:00  t(SATA)]  6 Gbps
  phy  26:U:attached:[5003048001f298da:00  t(SATA)]  6 Gbps
  phy  27:U:attached:[5003048001f298db:00  t(SATA)]  6 Gbps
  phy  28:U:attached:[5003048001f298dc:00  t(SATA)]  6 Gbps
  phy  29:U:attached:[5003048001f298dd:00  t(SATA)]  6 Gbps
  phy  31:U:attached:[5003048001f298df:00  t(SATA)]  6 Gbps
  phy  32:U:attached:[5003048001f298e0:00  t(SATA)]  6 Gbps
  phy  33:U:attached:[5003048001f298e1:00  t(SATA)]  6 Gbps
  phy  34:U:attached:[5003048001f298e2:00  t(SATA)]  6 Gbps
  phy  35:U:attached:[5003048001f298e3:00  t(SATA)]  6 Gbps
  phy  36:D:attached:[5003048001f298fd:00  V i(SSP+SMP) t(SSP)]  6 Gbps

Jeff Schaller · Accepted Answer · 2018-02-24 17:27:22Z

The very high SCSI device numbers (scsi 0:0:90903:0) show that there's a problem in this case that the hardware keeps dropping & re-initializing the drive.

The MPT SAS hardware does most of the re-initializing itself here, so we can't entirely control that from the Kernel. Separately, you mention having 21 drives, so they are probably behind one or more SAS expanders.

The question then becomes, it is possible, in software, to disable a port on a SAS expander?

If the expander actually supports it (I think it was optional in the standard), then yes!

The package in question is smp_utils. sg3_utils will also be helpful).

What you want is:

Figure out the expander device per the manpage above (probably ls /dev/bsg/expand*)
Confirm the faulty disks are attached to the phys from the dmesg: smp_discover /dev/bsg/expander-....
Disable the PHYs, in the form of smp_phy_control --phy=NN --op=di /dev/bsg/expander-.... Expanded for your case: smp_phy_control --phy=13 --op=di /dev/bsg/expander-0:0 smp_phy_control --phy=15 --op=di /dev/bsg/expander-0:0

The phy numbers were already in your output: 13, 15, but you might want to confirm them using smp_discover.

This is great info! Thanks. I added outputs from commands you mentioned above. — Mitar, Mar 20, 2017 at 21:57
Thanks for the output, I expanded the examples to what SHOULD work for you, assuming the PHYs match up. Did you run smp_discover? — robbat2, Mar 20, 2017 at 23:26
Ok, based on those, the smp_phy_control commands SHOULD be right, but the SAS ids seem to differ still, probably from flapping disks. — robbat2, Mar 21, 2017 at 9:59

Ed Neville · Accepted Answer · 2017-03-18 07:25:25Z

1

I think what you're trying to do sounds similar to the setup with Oracle ASM disks, where oracle accesses the blocks directly and the system operators are advised to exclude them using udev so the disks do not get formatted when in use by Oracle.

Here is a link to the related page on Oracle's site:

https://oracle-base.com/articles/linux/udev-scsi-rules-configuration-in-oracle-linux

answered Mar 18, 2017 at 7:25

Ed Neville

1,33010 silver badges11 bronze badges

No, as in the original question, the udev rules are just for mapping what devices appear as, and do not stop the kernel from getting devices from the controller.
– robbat2
Mar 18, 2017 at 20:38

Add a comment |

Stack Exchange Network

How to get Linux to completely ignore a SCSI drive?

2 Answers 2

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged
scsi
.

Linked

Hot Network Questions

How to get Linux to completely ignore a SCSI drive?

2 Answers 2

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged scsi.

Linked

Related

Hot Network Questions

Not the answer you're looking for? Browse other questions tagged
scsi
.