NVMe PCIe disk power cycling

Question

I want to test an NVMe SSD that is connected to a PCIe slot of my motherboard. The test procedure is a specific algorithm that writes workloads to the SSD, while the SSD is exposed to radiation (e.g., neutrons).

I am running Fedora 22, with kernel 4.4.6.

My current software successfully works with SATA SSD. Since the SSD can become unresponsive due to radiation, it's sometimes mandatory to power cycle it in order to resume operations. It is made possible with an externally controlled power supply.

Now, I would like to port my software to test NVMe SSD PCIe. I have modified a PCIe extender to externally apply voltage to the SSD; the derived power lines (+12V and 3.3V) are isolated from the PCIe connector power lines. With this setup, the SSD is well recognized – and works – when booting with the external power supply on.

Removing the device and re-scanning the PCI bus works as long as the NVMe SSD is powered on, namely:

echo 1 > /sys/bus/pci/devices/0000\:01\:00.0/remove

followed by:

echo 1 > /sys/bus/pci/rescan

works. However, if I power-off and then power-on the device after removing it, the PCI bus rescan does not work (and no message appears in dmesg).

If I "brutally" power off the SSD (with my controlled power supply) without removing the SSD under sysfs, I would get the following:

[  192.688934] nvme 0000:01:00.0: Failed status: ffffffff, reset controller
[  192.689274] Trying to free nonexistent resource <000000000000e000-000000000000e0ff>
[  192.699900] nvme 0000:01:00.0: Refused to change power state, currently in D3
[  192.699946] Trying to free nonexistent resource <000000000000e000-000000000000e0ff>
[  192.699953] nvme 0000:01:00.0: Device failed to resume

And obviously, rescanning the PCI bus does nothing.

Question: what would be necessary to achieve the power-cycling of the SSD without rebooting my test station? From similar threads, I understand that this problem is not trivial, so I would be content with a wide range of solutions (or hints), including:

• Adding kernel boot parameters
• Use of setpci commands (hints?)
• Use of extra logic, e.g., wire modifications on the PCIe extender to "fool" the PCIe bus
• Modifications in the kernel sources (hints?)

Answer 1

This is unlikely to succeed in getting the device to work again, but might get the device responsive enough to respond to the remove. Whilst the device is ok, save all the pci configuraton registers, and after the power-cycle restore them. You can get some way towards this by finding the controller slot

$ lspci | grep SATA
00:1f.2 SATA controller: Intel Corporation 7 Series Chipset Family 6-port SATA Controller [AHCI mode] (rev 04)

then listing the register names and passing each to setpci (you don't need to be root):

$ setpci --dumpregs |
awk -v slot='00:1f.2' 'NR>1 && !/ E?CAP/{
  reg = tolower($NF)
  printf "%s=",reg
  system("setpci -s " slot " " reg)
}'

This gets you lines like

vendor_id=8086
device_id=1e03
command=0407
status=02b0
base_address_0=0000f0b1
base_address_1=0000f0a1
base_address_2=0000f091
base_address_3=0000f081
base_address_4=0000f061
base_address_5=f7c06000

Obviously some of these registers are readonly, or have readonly bits. The idea is to call sudo setpci -s "$slot" with each of these lines, ignoring this aspect.

The above only handles the basic pci configuraton registers. However, you will need to save and restore some capability registers too. This will need more effort, depending on the register. You also need to be root to read them. For example,

sudo setpci -s 00:1f.2   CAP_MSI+0.l CAP_MSI+4.l CAP_MSI+8.l

will print the MSI capabilities registers:

00017005
fee0200c
000041b1

Compare these with the values shown by

sudo lspci -s "$slot" -vvv
    ...
    Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
            Address: fee0200c  Data: 41b1