How to Reset/Cycle Power to a PCIe Device?

Question

I have a PCIe device that only works correctly when the computer is fully powered off then on again. Issuing a simple reboot or reboot -p command does not appear to cycle the power to the PCIe card, which causes it not to work after the reboot.

Is there a way to, from the OS, cycle power to a device in a PCIe slot? I can find it in /sys/bus/pci/devices/0000*/, but I can't figure out how to properly reset the board. Toggling power seems to be the only way.

Barring that, can I change a setting somewhere that will cause a full power cycle on a reboot command?

I'm running Ubuntu 12.10 by the way.

Answer 1

Remove the device and rescan the PCIe bus

Removing the PCIe device via the remove function in its directory and reloading it via the PCIe bus' rescan function causes the kernel to power-cycle the PCIe device without rebooting your computer.

echo "1" > /sys/bus/pci/devices/DDDD\:BB\:DD.F/remove
sleep 1
echo "1" > /sys/bus/pci/rescan

In the above code, DDDD:BB:DD.F is the device slot format Domain:Bus:Device.Function.

Answer 2

Potential Method #1

I think you can do it with these commands:

disable

echo 0 > /sys/bus/pci/slots/$NUMBER/power

enable

echo 1 > /sys/bus/pci/slots/$NUMBER/power

Where $NUMBER is the number of the PCI slot.

lspci -vv may help to identify the device. This is not very well documented...

Potential Method #2

I came across this thread on U&L, similar issue: there are some answers to that question that say you can reset with this command:

echo "1" > /sys/bus/pci/devices/$NUMBER/reset

However, I would read the answers there! There are conditions around doing it this way! Specifically I would read this answer!

Potential Method #3

There is a Unix command, setpci, that may give you a method for resetting a device in the PCI bus.

I did not see any specific examples with this command so you'll have to google for examples and look through the man page. I would tread lightly with this command until you're confident in it's use. From what I've read about it, it's manipulating the hardware directly and so there are always risks in doing it yourself vs. using a tool that is exposing this type of functionality!

Answer 3

Resets in PCI express are a bit complex. There are two main types of resets - conventional reset, and function-level reset. There are also two types of conventional resets, fundamental resets and non-fundamental resets. See the PCI express specification for all of the details.

A 'cold reset' is a fundamental reset that takes place after power is applied to a PCIe device. There appears to be no standard way of triggering a cold reset, save for turning the system off and back on again. On my machines, the /sys/bus/pci/slots directory is empty.

A 'warm reset' is a fundamental reset that is triggered without disconnecting power from the device. There appears to be no standard way of triggering a warm reset.

A 'hot reset' is a conventional reset that is triggered across a PCI express link. A hot reset is triggered either when a link is forced into electrical idle or by sending TS1 and TS2 ordered sets with the hot reset bit set. Software can initiate a hot reset by setting and then clearing the secondary bus reset bit in the bridge control register in the PCI configuration space of the bridge port upstream of the device.

A 'function-level reset' (FLR) is a reset that affects only a single function of a PCI express device. It must not reset the entire PCIe device. Implementing function-level resets is not required by the PCIe specification. A function-level reset is initiated by setting the initiate function-level reset bit in the function's device control register in the PCI express capability structure in the PCI configuration space.

Linux exposes the function-level reset functionality in the form of /sys/bus/pci/devices/$dev/reset. Writing a 1 to this file will initiate a function-level reset on the corresponding function. Note that this only affects that specific function of the device, not the whole device, and devices are not required to implement function-level resets as per the PCIe specification.

I am not aware of any 'nice' method for triggering a hot reset (there is no sysfs entry for that). However, it is possible to use setpci to do so:

#!/bin/bash

dev=$1

if [ -z "$dev" ]; then
    echo "Error: no device specified"
    exit 1
fi

if [ ! -e "/sys/bus/pci/devices/$dev" ]; then
    dev="0000:$dev"
fi

if [ ! -e "/sys/bus/pci/devices/$dev" ]; then
    echo "Error: device $dev not found"
    exit 1
fi

port=$(basename $(dirname $(readlink "/sys/bus/pci/devices/$dev")))

if [ ! -e "/sys/bus/pci/devices/$port" ]; then
    echo "Error: device $port not found"
    exit 1
fi

echo "Removing $dev..."

echo 1 > "/sys/bus/pci/devices/$dev/remove"

echo "Performing hot reset of port $port..."

bc=$(setpci -s $port BRIDGE_CONTROL)

echo "Bridge control:" $bc

setpci -s $port BRIDGE_CONTROL=$(printf "%04x" $(("0x$bc" | 0x40)))
sleep 0.01
setpci -s $port BRIDGE_CONTROL=$bc
sleep 0.5

echo "Rescanning bus..."

echo 1 > "/sys/bus/pci/devices/$port/rescan"

Ensure that all attached drivers are unloaded before running this script. This script will attempt to remove the PCIe device, then command the upstream switch port to issue a hot reset, then attempt to rescan the PCIe bus. This script has also only been tested on devices with a single function, so it may need some reworking for devices with multiple functions.

Answer 4

Building off of the answer posted by alex.forencich

I had to make a few changes to get this working on CentOS 7, partly because I am not running as root. This version displays the commands being run.

#!/bin/bash

# e.g.  $ ./pcie_hot_reset.sh 04:00.0

DEV=$1

if [ -z "$DEV" ]; then
    echo "Error: no device specified"
    exit 1
fi

if [ ! -e "/sys/bus/pci/devices/$DEV" ]; then
    DEV="0000:$DEV"
fi

if [ ! -e "/sys/bus/pci/devices/$DEV" ]; then
    echo "Error: device $DEV not found"
    exit 1
fi

PORT=$(basename $(dirname $(readlink "/sys/bus/pci/devices/$DEV")))

if [ ! -e "/sys/bus/pci/devices/$PORT" ]; then
    echo "Error: device $PORT not found"
    exit 1
fi


echo -e "\nRemoving $DEV"

CMD="echo 1 | sudo tee /sys/bus/pci/devices/$DEV/remove"
printf "> $CMD\n"
eval $CMD


echo -e "\nPerforming hot reset of port $PORT"

CMD="setpci -s $PORT BRIDGE_CONTROL"
printf "> $CMD\n"
BR_CTRL=$(eval $CMD)

echo "Bridge control: $BR_CTRL"

CMD="sudo setpci -s $PORT BRIDGE_CONTROL=$(printf "%04x" $((0x${BR_CTRL} | 0x40)))"
printf "> $CMD\n"
eval $CMD
sleep 0.01

CMD="sudo setpci -s $PORT BRIDGE_CONTROL=$BR_CTRL"
printf "> $CMD\n"
eval $CMD
sleep 0.5


echo -e "\nRescanning bus"

CMD="echo 1 | sudo tee /sys/bus/pci/devices/$PORT/rescan"
printf "> $CMD\n"
eval $CMD

Sample output:

$ ./pcie_hot_reset.sh 04:00.0

Removing 0000:04:00.0
> echo 1 | sudo tee /sys/bus/pci/devices/0000:04:00.0/remove
1

Performing hot reset of port 0000:00:03.0
> setpci -s 0000:00:03.0 BRIDGE_CONTROL
Bridge control: 0010
> sudo setpci -s 0000:00:03.0 BRIDGE_CONTROL=0050
> sudo setpci -s 0000:00:03.0 BRIDGE_CONTROL=0010

Rescanning bus
> echo 1 | sudo tee /sys/bus/pci/devices/0000:00:03.0/rescan
1