47

I have a PCIe device that only works correctly when the computer is fully powered off then on again. Issuing a simple reboot or reboot -p command does not appear to cycle the power to the PCIe card, which causes it not to work after the reboot.

Is there a way to, from the OS, cycle power to a device in a PCIe slot? I can find it in /sys/bus/pci/devices/0000*/, but I can't figure out how to properly reset the board. Toggling power seems to be the only way.

Barring that, can I change a setting somewhere that will cause a full power cycle on a reboot command?

I'm running Ubuntu 12.10 by the way.

2
  • Did you try reboot -f? This is similar to pressing the power button of the CPU.
    – ktan
    Commented Apr 23, 2015 at 7:59
  • 3
    Two years ago the OP pointed out that a soft reboot didn't work. Your reboot -f is still a soft reboot. Commented Apr 23, 2015 at 10:18

4 Answers 4

39

Remove the device and rescan the PCIe bus

Removing the PCIe device via the remove function in its directory and reloading it via the PCIe bus' rescan function causes the kernel to power-cycle the PCIe device without rebooting your computer.

echo "1" > /sys/bus/pci/devices/DDDD\:BB\:DD.F/remove
sleep 1
echo "1" > /sys/bus/pci/rescan

In the above code, DDDD:BB:DD.F is the device slot format Domain:Bus:Device.Function.

5
  • echo "1" > /sys/bus/pci/rescan is worked for me in lenovo g560 mini pci slot. I plugged in USB 3.0 minipci card. System is Ubuntu 16.04 x64 Commented Mar 19, 2018 at 12:26
  • 1
    It doesn't work for all devices. I've got a Cavium network adapter that isn't power-off by that method as I can still access its u-boot when using a serial line.
    – Eric
    Commented Aug 13, 2019 at 9:53
  • This solution worked for me. I might suggest adding that you can use lspci to look for the Domain:Bus:Device:Function; I found mine listed as "Network controller" with a value of 00:14.3.
    – Andria
    Commented Aug 24, 2021 at 22:03
  • Worked on Xilinx ZCU106 with dual Samsung SSDs connected via the HPC0 port. Commented Apr 5, 2022 at 10:52
  • This sequence helped me to reset, in initramfs, a PCI NVMe SSD which was unable to enumerate because of the error "frozen state error detected, reset controller". Commented May 19, 2023 at 17:20
23

Resets in PCI express are a bit complex. There are two main types of resets - conventional reset, and function-level reset. There are also two types of conventional resets, fundamental resets and non-fundamental resets. See the PCI express specification for all of the details.

A 'cold reset' is a fundamental reset that takes place after power is applied to a PCIe device. There appears to be no standard way of triggering a cold reset, save for turning the system off and back on again. On my machines, the /sys/bus/pci/slots directory is empty.

A 'warm reset' is a fundamental reset that is triggered without disconnecting power from the device. There appears to be no standard way of triggering a warm reset.

A 'hot reset' is a conventional reset that is triggered across a PCI express link. A hot reset is triggered either when a link is forced into electrical idle or by sending TS1 and TS2 ordered sets with the hot reset bit set. Software can initiate a hot reset by setting and then clearing the secondary bus reset bit in the bridge control register in the PCI configuration space of the bridge port upstream of the device.

A 'function-level reset' (FLR) is a reset that affects only a single function of a PCI express device. It must not reset the entire PCIe device. Implementing function-level resets is not required by the PCIe specification. A function-level reset is initiated by setting the initiate function-level reset bit in the function's device control register in the PCI express capability structure in the PCI configuration space.

Linux exposes the function-level reset functionality in the form of /sys/bus/pci/devices/$dev/reset. Writing a 1 to this file will initiate a function-level reset on the corresponding function. Note that this only affects that specific function of the device, not the whole device, and devices are not required to implement function-level resets as per the PCIe specification.

I am not aware of any 'nice' method for triggering a hot reset (there is no sysfs entry for that). However, it is possible to use setpci to do so:

#!/bin/bash

dev=$1

if [ -z "$dev" ]; then
    echo "Error: no device specified"
    exit 1
fi

if [ ! -e "/sys/bus/pci/devices/$dev" ]; then
    dev="0000:$dev"
fi

if [ ! -e "/sys/bus/pci/devices/$dev" ]; then
    echo "Error: device $dev not found"
    exit 1
fi

port=$(basename $(dirname $(readlink "/sys/bus/pci/devices/$dev")))

if [ ! -e "/sys/bus/pci/devices/$port" ]; then
    echo "Error: device $port not found"
    exit 1
fi

echo "Removing $dev..."

echo 1 > "/sys/bus/pci/devices/$dev/remove"

echo "Performing hot reset of port $port..."

bc=$(setpci -s $port BRIDGE_CONTROL)

echo "Bridge control:" $bc

setpci -s $port BRIDGE_CONTROL=$(printf "%04x" $(("0x$bc" | 0x40)))
sleep 0.01
setpci -s $port BRIDGE_CONTROL=$bc
sleep 0.5

echo "Rescanning bus..."

echo 1 > "/sys/bus/pci/devices/$port/rescan"

Ensure that all attached drivers are unloaded before running this script. This script will attempt to remove the PCIe device, then command the upstream switch port to issue a hot reset, then attempt to rescan the PCIe bus. This script has also only been tested on devices with a single function, so it may need some reworking for devices with multiple functions.

13
  • This script worked for my AMD RX480. Context: PCI passthrough to a Win10 guest, then shutting down or rebooting the guest. Starting the guest again (without using this script) would hang if the GPU was still attached. Running this script in between fixed the issue
    – 小太郎
    Commented Oct 20, 2018 at 7:32
  • Is there any way to cut power to the PCIe device using this method? My gpu powers on with a full speed fan, and I'm looking for a way to simply cut power to it instead of loading the buggy amdgpu driver.
    – user128063
    Commented Aug 9, 2020 at 20:07
  • That is 100% dependent on the motherboard. There is no requirement in the PCIe spec to support anything like that that I am aware of, nor is there a standard API at the OS level. None of the machines I have used appear to have any way to actually cut power to a single PCIe slot. Commented Aug 9, 2020 at 20:12
  • @alex.forencich There is flaw in your script. The script removes a device, resets the bridge then rescans the bus. The flaw is that if the bridge is a switch (a bridge with multiple downstream ports) then the reset thereof will propagate to all downstream devices. Also, the reset might affect the bridge itself. The solution is to adjust your script such that it removes all immediate (nearest child) downstream devices instead of only removing $dev, then perform the reset, then remove the bridge, then rescan. Commented Jan 5, 2022 at 10:35
  • 1
    0x40 is bit 6 in the bridge control register, which is the secondary bus reset bit. It's documented in the PCIe specification. In the gen 3 spec, this is on page 600, in table 7-6 in section 7.5.3.6. Commented Mar 15, 2022 at 22:01
20

Potential Method #1

I think you can do it with these commands:

disable

echo 0 > /sys/bus/pci/slots/$NUMBER/power

enable

echo 1 > /sys/bus/pci/slots/$NUMBER/power

Where $NUMBER is the number of the PCI slot.

lspci -vv may help to identify the device. This is not very well documented...

Potential Method #2

I came across this thread on U&L, similar issue: there are some answers to that question that say you can reset with this command:

echo "1" > /sys/bus/pci/devices/$NUMBER/reset

However, I would read the answers there! There are conditions around doing it this way! Specifically I would read this answer!

Potential Method #3

There is a Unix command, setpci, that may give you a method for resetting a device in the PCI bus.

I did not see any specific examples with this command so you'll have to google for examples and look through the man page. I would tread lightly with this command until you're confident in it's use. From what I've read about it, it's manipulating the hardware directly and so there are always risks in doing it yourself vs. using a tool that is exposing this type of functionality!

22
  • 3
    Nothing shows up in slots, even though I have multiple cards plugged in. I do have a power directory in /sys/bus/pci/devices/$NUMBER/. But nothing seems to warrant setting 0 or 1
    – zachd1_618
    Commented Apr 27, 2013 at 23:30
  • 1
    I came across this thread on U&L, similar issue: there are some answers to that Q that say you can reset with this: echo "1" > /sys/bus/pci/devices/$NUMBER/reset. Read that Q though, there are conditions around doing it that way!
    – slm
    Commented Apr 27, 2013 at 23:42
  • 1
    Are you unloading the kernel modules for that card prior to the power cycling? I think you have to do that as well.
    – slm
    Commented Apr 28, 2013 at 0:33
  • 1
    I think I'll check the kernel source code to see if toggling power actually puts it into D3.
    – forest
    Commented Aug 7, 2018 at 0:04
  • 1
    'power' should put the card in D3cold by removing power to the card, but support for this at the motherboard level is uncommon. 'reset' issues an FLR to one function and will not change the card power state. Commented Apr 12, 2019 at 22:57
1

Building off of the answer posted by alex.forencich

I had to make a few changes to get this working on CentOS 7, partly because I am not running as root. This version displays the commands being run.

#!/bin/bash

# e.g.  $ ./pcie_hot_reset.sh 04:00.0

DEV=$1

if [ -z "$DEV" ]; then
    echo "Error: no device specified"
    exit 1
fi

if [ ! -e "/sys/bus/pci/devices/$DEV" ]; then
    DEV="0000:$DEV"
fi

if [ ! -e "/sys/bus/pci/devices/$DEV" ]; then
    echo "Error: device $DEV not found"
    exit 1
fi

PORT=$(basename $(dirname $(readlink "/sys/bus/pci/devices/$DEV")))

if [ ! -e "/sys/bus/pci/devices/$PORT" ]; then
    echo "Error: device $PORT not found"
    exit 1
fi


echo -e "\nRemoving $DEV"

CMD="echo 1 | sudo tee /sys/bus/pci/devices/$DEV/remove"
printf "> $CMD\n"
eval $CMD


echo -e "\nPerforming hot reset of port $PORT"

CMD="setpci -s $PORT BRIDGE_CONTROL"
printf "> $CMD\n"
BR_CTRL=$(eval $CMD)

echo "Bridge control: $BR_CTRL"

CMD="sudo setpci -s $PORT BRIDGE_CONTROL=$(printf "%04x" $((0x${BR_CTRL} | 0x40)))"
printf "> $CMD\n"
eval $CMD
sleep 0.01

CMD="sudo setpci -s $PORT BRIDGE_CONTROL=$BR_CTRL"
printf "> $CMD\n"
eval $CMD
sleep 0.5


echo -e "\nRescanning bus"

CMD="echo 1 | sudo tee /sys/bus/pci/devices/$PORT/rescan"
printf "> $CMD\n"
eval $CMD

Sample output:

$ ./pcie_hot_reset.sh 04:00.0

Removing 0000:04:00.0
> echo 1 | sudo tee /sys/bus/pci/devices/0000:04:00.0/remove
1

Performing hot reset of port 0000:00:03.0
> setpci -s 0000:00:03.0 BRIDGE_CONTROL
Bridge control: 0010
> sudo setpci -s 0000:00:03.0 BRIDGE_CONTROL=0050
> sudo setpci -s 0000:00:03.0 BRIDGE_CONTROL=0010

Rescanning bus
> echo 1 | sudo tee /sys/bus/pci/devices/0000:00:03.0/rescan
1
4
  • What's the advantage of filling the script with a bunch of "sudo" instead of just running the whole script with sudo? Commented Mar 18, 2020 at 2:08
  • There is no advantage. It is just a style that often gets used where I work.
    – mhck
    Commented Mar 18, 2020 at 10:28
  • One obvious advantage is that you can run the script as a user. Commented Apr 17, 2023 at 11:58
  • there is not rescan file per port, are you sure this is going to work for current kernel?
    – Wang
    Commented Sep 8, 2023 at 9:59

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .