38

I have a PCIe device that only works correctly when the computer is fully powered off then on again. Issuing a simple reboot or reboot -p command does not appear to cycle the power to the PCIe card, which causes it not to work after the reboot.

Is there a way to, from the OS, cycle power to a device in a PCIe slot? I can find it in /sys/bus/pci/devices/0000*/, but I can't figure out how to properly reset the board. Toggling power seems to be the only way.

Barring that, can I change a setting somewhere that will cause a full power cycle on a reboot command?

I'm running Ubuntu 12.10 by the way.

2
  • Did you try reboot -f? This is similar to pressing the power button of the CPU.
    – ktan
    Apr 23, 2015 at 7:59
  • 3
    Two years ago the OP pointed out that a soft reboot didn't work. Your reboot -f is still a soft reboot.
    – roaima
    Apr 23, 2015 at 10:18

4 Answers 4

24

Remove the device and rescan the PCIe bus

Removing the PCIe device via the remove function in its directory and reloading it via the PCIe bus' rescan function causes the kernel to power-cycle the PCIe device without rebooting your computer.

echo "1" > /sys/bus/pci/devices/DDDD\:BB\:DD.F/remove
sleep 1
echo "1" > /sys/bus/pci/rescan

In the above code, DDDD:BB:DD.F is the device slot format Domain:Bus:Device.Function.

4
  • echo "1" > /sys/bus/pci/rescan is worked for me in lenovo g560 mini pci slot. I plugged in USB 3.0 minipci card. System is Ubuntu 16.04 x64 Mar 19, 2018 at 12:26
  • It doesn't work for all devices. I've got a Cavium network adapter that isn't power-off by that method as I can still access its u-boot when using a serial line.
    – Eric
    Aug 13, 2019 at 9:53
  • This solution worked for me. I might suggest adding that you can use lspci to look for the Domain:Bus:Device:Function; I found mine listed as "Network controller" with a value of 00:14.3.
    – Andria
    Aug 24, 2021 at 22:03
  • Worked on Xilinx ZCU106 with dual Samsung SSDs connected via the HPC0 port. Apr 5 at 10:52
19

Potential Method #1

I think you can do it with these commands:

disable

echo 0 > /sys/bus/pci/slots/$NUMBER/power

enable

echo 1 > /sys/bus/pci/slots/$NUMBER/power

Where $NUMBER is the number of the PCI slot.

lspci -vv may help to identify the device. This is not very well documented...

Potential Method #2

I came across this thread on U&L, similar issue: there are some answers to that question that say you can reset with this command:

echo "1" > /sys/bus/pci/devices/$NUMBER/reset

However, I would read the answers there! There are conditions around doing it this way! Specifically I would read this answer!

Potential Method #3

There is a Unix command, setpci, that may give you a method for resetting a device in the PCI bus.

I did not see any specific examples with this command so you'll have to google for examples and look through the man page. I would tread lightly with this command until you're confident in it's use. From what I've read about it, it's manipulating the hardware directly and so there are always risks in doing it yourself vs. using a tool that is exposing this type of functionality!

21
  • 1
    Nothing shows up in slots, even though I have multiple cards plugged in. I do have a power directory in /sys/bus/pci/devices/$NUMBER/. But nothing seems to warrant setting 0 or 1
    – zachd1_618
    Apr 27, 2013 at 23:30
  • 1
    I came across this thread on U&L, similar issue: there are some answers to that Q that say you can reset with this: echo "1" > /sys/bus/pci/devices/$NUMBER/reset. Read that Q though, there are conditions around doing it that way!
    – slm
    Apr 27, 2013 at 23:42
  • 1
    Are you unloading the kernel modules for that card prior to the power cycling? I think you have to do that as well.
    – slm
    Apr 28, 2013 at 0:33
  • 1
    I think I'll check the kernel source code to see if toggling power actually puts it into D3.
    – forest
    Aug 7, 2018 at 0:04
  • 1
    'power' should put the card in D3cold by removing power to the card, but support for this at the motherboard level is uncommon. 'reset' issues an FLR to one function and will not change the card power state. Apr 12, 2019 at 22:57
16

Resets in PCI express are a bit complex. There are two main types of resets - conventional reset, and function-level reset. There are also two types of conventional resets, fundamental resets and non-fundamental resets. See the PCI express specification for all of the details.

A 'cold reset' is a fundamental reset that takes place after power is applied to a PCIe device. There appears to be no standard way of triggering a cold reset, save for turning the system off and back on again. On my machines, the /sys/bus/pci/slots directory is empty.

A 'warm reset' is a fundamental reset that is triggered without disconnecting power from the device. There appears to be no standard way of triggering a warm reset.

A 'hot reset' is a conventional reset that is triggered across a PCI express link. A hot reset is triggered either when a link is forced into electrical idle or by sending TS1 and TS2 ordered sets with the hot reset bit set. Software can initiate a hot reset by setting and then clearing the secondary bus reset bit in the bridge control register in the PCI configuration space of the bridge port upstream of the device.

A 'function-level reset' (FLR) is a reset that affects only a single function of a PCI express device. It must not reset the entire PCIe device. Implementing function-level resets is not required by the PCIe specification. A function-level reset is initiated by setting the initiate function-level reset bit in the function's device control register in the PCI express capability structure in the PCI configuration space.

Linux exposes the function-level reset functionality in the form of /sys/bus/pci/devices/$dev/reset. Writing a 1 to this file will initiate a function-level reset on the corresponding function. Note that this only affects that specific function of the device, not the whole device, and devices are not required to implement function-level resets as per the PCIe specification.

I am not aware of any 'nice' method for triggering a hot reset (there is no sysfs entry for that). However, it is possible to use setpci to do so:

#!/bin/bash

dev=$1

if [ -z "$dev" ]; then
    echo "Error: no device specified"
    exit 1
fi

if [ ! -e "/sys/bus/pci/devices/$dev" ]; then
    dev="0000:$dev"
fi

if [ ! -e "/sys/bus/pci/devices/$dev" ]; then
    echo "Error: device $dev not found"
    exit 1
fi

port=$(basename $(dirname $(readlink "/sys/bus/pci/devices/$dev")))

if [ ! -e "/sys/bus/pci/devices/$port" ]; then
    echo "Error: device $port not found"
    exit 1
fi

echo "Removing $dev..."

echo 1 > "/sys/bus/pci/devices/$dev/remove"

echo "Performing hot reset of port $port..."

bc=$(setpci -s $port BRIDGE_CONTROL)

echo "Bridge control:" $bc

setpci -s $port BRIDGE_CONTROL=$(printf "%04x" $(("0x$bc" | 0x40)))
sleep 0.01
setpci -s $port BRIDGE_CONTROL=$bc
sleep 0.5

echo "Rescanning bus..."

echo 1 > "/sys/bus/pci/devices/$port/rescan"

Ensure that all attached drivers are unloaded before running this script. This script will attempt to remove the PCIe device, then command the upstream switch port to issue a hot reset, then attempt to rescan the PCIe bus. This script has also only been tested on devices with a single function, so it may need some reworking for devices with multiple functions.

13
  • This script worked for my AMD RX480. Context: PCI passthrough to a Win10 guest, then shutting down or rebooting the guest. Starting the guest again (without using this script) would hang if the GPU was still attached. Running this script in between fixed the issue
    – 小太郎
    Oct 20, 2018 at 7:32
  • Is there any way to cut power to the PCIe device using this method? My gpu powers on with a full speed fan, and I'm looking for a way to simply cut power to it instead of loading the buggy amdgpu driver.
    – user128063
    Aug 9, 2020 at 20:07
  • That is 100% dependent on the motherboard. There is no requirement in the PCIe spec to support anything like that that I am aware of, nor is there a standard API at the OS level. None of the machines I have used appear to have any way to actually cut power to a single PCIe slot. Aug 9, 2020 at 20:12
  • @alex.forencich There is flaw in your script. The script removes a device, resets the bridge then rescans the bus. The flaw is that if the bridge is a switch (a bridge with multiple downstream ports) then the reset thereof will propagate to all downstream devices. Also, the reset might affect the bridge itself. The solution is to adjust your script such that it removes all immediate (nearest child) downstream devices instead of only removing $dev, then perform the reset, then remove the bridge, then rescan. Jan 5 at 10:35
  • 1
    0x40 is bit 6 in the bridge control register, which is the secondary bus reset bit. It's documented in the PCIe specification. In the gen 3 spec, this is on page 600, in table 7-6 in section 7.5.3.6. Mar 15 at 22:01
0

Building off of the answer posted by alex.forencich

I had to make a few changes to get this working on CentOS 7, partly because I am not running as root. This version displays the commands being run.

#!/bin/bash

# e.g.  $ ./pcie_hot_reset.sh 04:00.0

DEV=$1

if [ -z "$DEV" ]; then
    echo "Error: no device specified"
    exit 1
fi

if [ ! -e "/sys/bus/pci/devices/$DEV" ]; then
    DEV="0000:$DEV"
fi

if [ ! -e "/sys/bus/pci/devices/$DEV" ]; then
    echo "Error: device $DEV not found"
    exit 1
fi

PORT=$(basename $(dirname $(readlink "/sys/bus/pci/devices/$DEV")))

if [ ! -e "/sys/bus/pci/devices/$PORT" ]; then
    echo "Error: device $PORT not found"
    exit 1
fi


echo -e "\nRemoving $DEV"

CMD="echo 1 | sudo tee /sys/bus/pci/devices/$DEV/remove"
printf "> $CMD\n"
eval $CMD


echo -e "\nPerforming hot reset of port $PORT"

CMD="setpci -s $PORT BRIDGE_CONTROL"
printf "> $CMD\n"
BR_CTRL=$(eval $CMD)

echo "Bridge control: $BR_CTRL"

CMD="sudo setpci -s $PORT BRIDGE_CONTROL=$(printf "%04x" $((0x${BR_CTRL} | 0x40)))"
printf "> $CMD\n"
eval $CMD
sleep 0.01

CMD="sudo setpci -s $PORT BRIDGE_CONTROL=$BR_CTRL"
printf "> $CMD\n"
eval $CMD
sleep 0.5


echo -e "\nRescanning bus"

CMD="echo 1 | sudo tee /sys/bus/pci/devices/$PORT/rescan"
printf "> $CMD\n"
eval $CMD

Sample output:

$ ./pcie_hot_reset.sh 04:00.0

Removing 0000:04:00.0
> echo 1 | sudo tee /sys/bus/pci/devices/0000:04:00.0/remove
1

Performing hot reset of port 0000:00:03.0
> setpci -s 0000:00:03.0 BRIDGE_CONTROL
Bridge control: 0010
> sudo setpci -s 0000:00:03.0 BRIDGE_CONTROL=0050
> sudo setpci -s 0000:00:03.0 BRIDGE_CONTROL=0010

Rescanning bus
> echo 1 | sudo tee /sys/bus/pci/devices/0000:00:03.0/rescan
1
2
  • What's the advantage of filling the script with a bunch of "sudo" instead of just running the whole script with sudo? Mar 18, 2020 at 2:08
  • There is no advantage. It is just a style that often gets used where I work.
    – mhck
    Mar 18, 2020 at 10:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.