9

I got an IBM x3850 type 8864 machine, I can successfully boot using a 2.6.32 kernel but when I try to use a 3.10 kernel or newer the kernel fails to initialize all PCI slots (I can fix this (manually), see below):

pci 0000:19:00.0: BAR 14: can't assign mem (size 0x1a00000)
pci 0000:19:00.0: BAR 13: can't assign io (size 0x3000)
pci 0000:19:00.0: BAR 14: can't assign mem (size 0x1600000)
pci 0000:19:00.0: BAR 13: can't assign io (size 0x3000)
pci 0000:1a:00.0: BAR 14: can't assign mem (size 0x1600000)
pci 0000:1a:00.0: BAR 13: assigned [io  0x7000-0x8fff]
pci 0000:1b:02.0: BAR 14: can't assign mem (size 0xa00000)
pci 0000:1b:04.0: BAR 14: can't assign mem (size 0xa00000)
pci 0000:1b:02.0: BAR 13: assigned [io  0x7000-0x7fff]
pci 0000:1b:04.0: BAR 13: assigned [io  0x8000-0x8fff]
...

Which causes that my network card is not successfully loaded as the PCI bus is obviously not correctly instantiated.

lspci yields the following:

00:00.0 Host bridge: IBM Calgary PCI-X Host Bridge (rev 04)
00:01.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] RV100 [Radeon 7000 / Radeon VE]
00:03.0 USB controller: NEC Corporation OHCI USB Controller (rev 43)
00:03.1 USB controller: NEC Corporation OHCI USB Controller (rev 43)
00:03.2 USB controller: NEC Corporation uPD72010x USB 2.0 Controller (rev 04)
00:0f.0 Host bridge: Broadcom CSB6 South Bridge (rev a0)
00:0f.1 IDE interface: Broadcom CSB6 RAID/IDE Controller (rev a0)
00:0f.3 ISA bridge: Broadcom GCLE-2 Host Bridge
01:00.0 Host bridge: IBM Calgary PCI-X Host Bridge (rev 04)
01:01.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5704 Gigabit Ethernet (rev 10)
01:01.1 Ethernet controller: Broadcom Corporation NetXtreme BCM5704 Gigabit Ethernet (rev 10)
01:02.0 RAID bus controller: Adaptec AAC-RAID (rev 02)
02:00.0 Host bridge: IBM Calgary PCI-X Host Bridge (rev 04)
06:00.0 Host bridge: IBM Calgary PCI-X Host Bridge (rev 04)
0a:00.0 PCI bridge: IBM CalIOC2 PCI-E Root Port (rev 01)
0f:00.0 PCI bridge: IBM CalIOC2 PCI-E Root Port (rev 01)
14:00.0 PCI bridge: IBM CalIOC2 PCI-E Root Port (rev 01)
19:00.0 PCI bridge: IBM CalIOC2 PCI-E Root Port (rev 01)
1a:00.0 PCI bridge: Integrated Device Technology, Inc. [IDT] PES12N3A PCI Express Switch (rev 0c)
1b:02.0 PCI bridge: Integrated Device Technology, Inc. [IDT] PES12N3A PCI Express Switch (rev 0c)
1b:04.0 PCI bridge: Integrated Device Technology, Inc. [IDT] PES12N3A PCI Express Switch (rev 0c)
1c:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
1c:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
1d:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
1d:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)

FIX

I can actually fix it by removing the root PCI bus 19:00.0 PCI bridge: IBM CalIOC2 PCI-E Root Port (rev 01) via echo 1 > /sys/bus/pci/devices/0000\:19\:00.0/remove and a rescan afterwards: echo 1 > /sys/bus/pci/rescan which causes the following output:

pci _bus 0000:1c: busn_res: [bus 1c] is released
pci _bus 0000:1d: busn_res: [bus 1d] is released
pci _bus 0000:1b: busn_res: [bus 1b-1d] is released
pci _bus 0000:1a: busn_res: [bus 1a-1d] is released
pci  0000:19:00.0: [1014:0308] type 01 class 0x060401
pci  0000:19:00.0: supports D1 D2
pci  0000:19:00.0: PME# supported from D0 D1 D2 D3hot D3cold
pci  0000:1a:00.0: [111d:8018] type 01 class 0x060400
pci  0000:1a:00.0: PME# supported from D0 D3hot D3cold
pci  0000:19:00.0:     pci  bridge to [bus 1a-1d] (subtractive decode)
pci  0000:19:00.0:   bridge window [mem 0xea800000-0xea9fffff 64bit pref]
pci  0000:19:00.0:   bridge window [mem 0xea800000-0xebcfffff] (subtractive decode)
pci  0000:19:00.0:   bridge window [io  0x7000-0x8fff] (subtractive decode)
pci  0000:1b:02.0: [111d:8018] type 01 class 0x060400
pci  0000:1b:02.0: PME# supported from D0 D3hot D3cold
pci  0000:1b:04.0: [111d:8018] type 01 class 0x060400
pci  0000:1b:04.0: PME# supported from D0 D3hot D3cold
pci  0000:1a:00.0:     pci  bridge to [bus 1b-1d]
pci  0000:1a:00.0:   bridge window [io  0x7000-0x8fff]
pci  0000:1a:00.0:   bridge window [mem 0xea800000-0xea9fffff 64bit pref]

....

pci  0000:1b:04.0:   bridge window [mem 0xea900000-0xea9fffff 64bit pref]
pci  0000:1b:02.0: bridge window [mem 0x00100000-0x001fffff 64bit pref] to [bus 1c] add_size 100000
pci  0000:1b:04.0: bridge window [mem 0x00100000-0x001fffff 64bit pref] to [bus 1d] add_size 100000
pci  0000:1b:02.0: res[15]=[mem 0x00100000-0x001fffff 64bit pref] get_res_add_size add_size 100000
pci  0000:1b:04.0: res[15]=[mem 0x00100000-0x001fffff 64bit pref] get_res_add_size add_size 100000
pci  0000:1a:00.0: bridge window [mem 0x00100000-0x002fffff 64bit pref] to [bus 1b-1d] add_size 200000
pci  0000:19:00.0: bridge window [io  0x1000-0x2fff] to [bus 1a-1d] add_size 1000
pci  0000:1a:00.0: res[15]=[mem 0x00100000-0x002fffff 64bit pref] get_res_add_size add_size 200000
pci  0000:19:00.0: bridge window [mem 0x00100000-0x002fffff 64bit pref] to [bus 1a-1d] add_size 200000
pci  0000:19:00.0: bridge window [mem 0x00200000-0x015fffff] to [bus 1a-1d] add_size 200000
pci  0000:19:00.0: res[14]=[mem 0x00200000-0x015fffff] get_res_add_size add_size 200000
pci  0000:19:00.0: res[15]=[mem 0x00100000-0x002fffff 64bit pref] get_res_add_size add_size 200000
pci  0000:19:00.0: res[13]=[io  0x1000-0x2fff] get_res_add_size add_size 1000
pci  0000:19:00.0: BAR 14: can't assign mem (size 0x1600000)
pci  0000:19:00.0: BAR 15: assigned [mem 0xea800000-0xeabfffff 64bit pref]
pci  0000:19:00.0: BAR 13: can't assign io (size 0x3000)
pci  0000:19:00.0: BAR 14: assigned [mem 0xea800000-0xebbfffff]
pci  0000:19:00.0: BAR 15: can't assign mem pref (size 0x200000)
pci  0000:19:00.0: BAR 13: assigned [io  0x7000-0x8fff]
pci  0000:19:00.0: BAR 14: can't assign mem (size 0x1400000)
pci  0000:19:00.0: failed to add 200000 res[14]=[mem 0xea800000-0xebbfffff]
pci  0000:19:00.0: BAR 13: can't assign io (size 0x2000)
pci  0000:19:00.0: failed to add 1000 res[13]=[io  0x7000-0x8fff]
pci  0000:1a:00.0: res[15]=[mem 0x00100000-0x002fffff 64bit pref] get_res_add_size add_size 200000
pci  0000:1a:00.0: BAR 14: assigned [mem 0xea800000-0xebbfffff]
pci  0000:1a:00.0: BAR 15: can't assign mem pref (size 0x400000)
pci  0000:1a:00.0: BAR 13: assigned [io  0x7000-0x8fff]
pci  0000:1a:00.0: BAR 14: assigned [mem 0xea800000-0xebbfffff]
pci  0000:1a:00.0: BAR 15: can't assign mem pref (size 0x200000)
pci  0000:1b:02.0: res[15]=[mem 0x00100000-0x001fffff 64bit pref] get_res_add_size add_size 100000
pci  0000:1b:04.0: res[15]=[mem 0x00100000-0x001fffff 64bit pref] get_res_add_size add_size 100000
pci  0000:1b:02.0: BAR 14: assigned [mem 0xea800000-0xeb1fffff]
pci  0000:1b:04.0: BAR 14: assigned [mem 0xeb200000-0xebbfffff]
pci  0000:1b:02.0: BAR 15: can't assign mem pref (size 0x200000)
pci  0000:1b:04.0: BAR 15: can't assign mem pref (size 0x200000)
pci  0000:1b:02.0: BAR 13: assigned [io  0x7000-0x7fff]
pci  0000:1b:04.0: BAR 13: assigned [io  0x8000-0x8fff]
pci  0000:1b:02.0: BAR 14: assigned [mem 0xea800000-0xeb1fffff]
pci  0000:1b:04.0: BAR 14: assigned [mem 0xeb200000-0xebbfffff]
pci  0000:1b:02.0: BAR 15: can't assign mem pref (size 0x100000)
pci  0000:1b:04.0: BAR 15: can't assign mem pref (size 0x100000)
pci  0000:1c:00.0: reg 184: [mem 0xea800000-0xea803fff 64bit pref]
pci  0000:1c:00.0: reg 190: [mem 0xea820000-0xea823fff 64bit pref]
pci  0000:1c:00.0: reg 184: [mem 0xea800000-0xea803fff 64bit pref]
pci  0000:1c:00.0: reg 184: [mem 0xea800000-0xea803fff 64bit pref]
pci  0000:1c:00.0: reg 190: [mem 0xea820000-0xea823fff 64bit pref]
pci  0000:1c:00.0: reg 184: [mem 0xea800000-0xea803fff 64bit pref]
pci  0000:1c:00.0: reg 190: [mem 0xea820000-0xea823fff 64bit pref]
pci  0000:1c:00.1: reg 184: [mem 0xea840000-0xea843fff 64bit pref]
pci  0000:1c:00.0: reg 184: [mem 0xea800000-0xea803fff 64bit pref]
pci  0000:1c:00.0: reg 190: [mem 0xea820000-0xea823fff 64bit pref]
pci  0000:1c:00.1: reg 190: [mem 0xea860000-0xea863fff 64bit pref]
pci  0000:1c:00.0: reg 184: [mem 0xea800000-0xea803fff 64bit pref]
pci  0000:1c:00.0: reg 190: [mem 0xea820000-0xea823fff 64bit pref]
pci  0000:1c:00.1: reg 184: [mem 0xea840000-0xea843fff 64bit pref]
pci  0000:1c:00.0: res[7]=[mem 0xea800000-0xea7fffff 64bit pref] get_res_add_size add_size 20000
pci  0000:1c:00.0: res[10]=[mem 0xea820000-0xea81ffff 64bit pref] get_res_add_size add_size 20000
pci  0000:1c:00.1: res[7]=[mem 0xea840000-0xea83ffff 64bit pref] get_res_add_size add_size 20000
pci  0000:1c:00.1: res[10]=[mem 0xea860000-0xea85ffff 64bit pref] get_res_add_size add_size 20000
pci  0000:1c:00.0: BAR 1: assigned [mem 0xea800000-0xeabfffff]
pci  0000:1c:00.1: BAR 1: assigned [mem 0xeac00000-0xeaffffff]
pci  0000:1c:00.0: BAR 0: assigned [mem 0xeb000000-0xeb01ffff]
pci  0000:1c:00.1: BAR 0: assigned [mem 0xeb020000-0xeb03ffff]
pci  0000:1c:00.0: BAR 3: assigned [mem 0xeb040000-0xeb043fff]
pci  0000:1c:00.0: reg 184: [mem 0xea800000-0xea803fff 64bit pref]
pci  0000:1c:00.0: BAR 7: assigned [mem 0xeb044000-0xeb063fff 64bit pref]
pci  0000:1c:00.0: reg 190: [mem 0xea820000-0xea823fff 64bit pref]
pci  0000:1c:00.0: BAR 10: assigned [mem 0xeb064000-0xeb083fff 64bit pref]
pci  0000:1c:00.1: BAR 3: assigned [mem 0xeb084000-0xeb087fff]
pci  0000:1c:00.1: reg 184: [mem 0xea840000-0xea843fff 64bit pref]
pci  0000:1c:00.1: BAR 7: assigned [mem 0xeb088000-0xeb0a7fff 64bit pref]
pci  0000:1c:00.1: reg 190: [mem 0xea860000-0xea863fff 64bit pref]
pci  0000:1c:00.1: BAR 10: assigned [mem 0xeb0a8000-0xeb0c7fff 64bit pref]
pci  0000:1c:00.0: BAR 2: assigned [io  0x7000-0x701f]
pci  0000:1c:00.1: BAR 2: assigned [io  0x7020-0x703f]
pci  0000:1b:02.0:     pci  bridge to [bus 1c]
pci  0000:1b:02.0:   bridge window [io  0x7000-0x7fff]
pci  0000:1b:02.0:   bridge window [mem 0xea800000-0xeb1fffff]
pci  0000:1d:00.0: reg 184: [mem 0xea900000-0xea903fff 64bit pref]
pci  0000:1d:00.0: reg 190: [mem 0xea920000-0xea923fff 64bit pref]
pci  0000:1d:00.0: reg 184: [mem 0xea900000-0xea903fff 64bit pref]
pci  0000:1d:00.0: reg 184: [mem 0xea900000-0xea903fff 64bit pref]
pci  0000:1d:00.0: reg 190: [mem 0xea920000-0xea923fff 64bit pref]
pci  0000:1d:00.0: reg 184: [mem 0xea900000-0xea903fff 64bit pref]
pci  0000:1d:00.0: reg 190: [mem 0xea920000-0xea923fff 64bit pref]
pci  0000:1d:00.1: reg 184: [mem 0xea940000-0xea943fff 64bit pref]
pci  0000:1d:00.0: reg 184: [mem 0xea900000-0xea903fff 64bit pref]
pci  0000:1d:00.0: reg 190: [mem 0xea920000-0xea923fff 64bit pref]
pci  0000:1d:00.1: reg 190: [mem 0xea960000-0xea963fff 64bit pref]
pci  0000:1d:00.0: reg 184: [mem 0xea900000-0xea903fff 64bit pref]
pci  0000:1d:00.0: reg 190: [mem 0xea920000-0xea923fff 64bit pref]
pci  0000:1d:00.1: reg 184: [mem 0xea940000-0xea943fff 64bit pref]
pci  0000:1d:00.0: res[7]=[mem 0xea900000-0xea8fffff 64bit pref] get_res_add_size add_size 20000
pci  0000:1d:00.0: res[10]=[mem 0xea920000-0xea91ffff 64bit pref] get_res_add_size add_size 20000
pci  0000:1d:00.1: res[7]=[mem 0xea940000-0xea93ffff 64bit pref] get_res_add_size add_size 20000
pci  0000:1d:00.1: res[10]=[mem 0xea960000-0xea95ffff 64bit pref] get_res_add_size add_size 20000
pci  0000:1d:00.0: BAR 1: assigned [mem 0xeb400000-0xeb7fffff]
pci  0000:1d:00.1: BAR 1: assigned [mem 0xeb800000-0xebbfffff]
pci  0000:1d:00.0: BAR 0: assigned [mem 0xeb200000-0xeb21ffff]
pci  0000:1d:00.1: BAR 0: assigned [mem 0xeb220000-0xeb23ffff]
pci  0000:1d:00.0: BAR 3: assigned [mem 0xeb240000-0xeb243fff]
pci  0000:1d:00.0: reg 184: [mem 0xea900000-0xea903fff 64bit pref]
pci  0000:1d:00.0: BAR 7: assigned [mem 0xeb244000-0xeb263fff 64bit pref]
pci  0000:1d:00.0: reg 190: [mem 0xea920000-0xea923fff 64bit pref]
pci  0000:1d:00.0: BAR 10: assigned [mem 0xeb264000-0xeb283fff 64bit pref]
pci  0000:1d:00.1: BAR 3: assigned [mem 0xeb284000-0xeb287fff]
pci  0000:1d:00.1: reg 184: [mem 0xea940000-0xea943fff 64bit pref]
pci  0000:1d:00.1: BAR 7: assigned [mem 0xeb288000-0xeb2a7fff 64bit pref]
pci  0000:1d:00.1: reg 190: [mem 0xea960000-0xea963fff 64bit pref]
pci  0000:1d:00.1: BAR 10: assigned [mem 0xeb2a8000-0xeb2c7fff 64bit pref]
pci  0000:1d:00.0: BAR 2: assigned [io  0x8000-0x801f]
pci  0000:1d:00.1: BAR 2: assigned [io  0x8020-0x803f]
pci  0000:1b:04.0:     pci  bridge to [bus 1d]
pci  0000:1b:04.0:   bridge window [io  0x8000-0x8fff]
pci  0000:1b:04.0:   bridge window [mem 0xeb200000-0xebbfffff]
pci  0000:1a:00.0:     pci  bridge to [bus 1b-1d]
pci  0000:1a:00.0:   bridge window [io  0x7000-0x8fff]
pci  0000:1a:00.0:   bridge window [mem 0xea800000-0xebbfffff]
pci  0000:19:00.0:     pci  bridge to [bus 1a-1d]
pci  0000:19:00.0:   bridge window [io  0x7000-0x8fff]
pci  0000:19:00.0:   bridge window [mem 0xea800000-0xebbfffff]

QUESTION

Is it somehow possible to tell the kernel (e.g. via a parameter) to automatically do this? What is causing this issue in the first place?

Thank you in advance!

Update

As the described fix fails on a 4.x system (actually starting with 3.12 I suppose), I had a look at the kernel and found that if I disable PCI ASPM (which was already disabled by ACPI (t can also be forced by pcie_aspm=off in kernel boot parameter)), the following small fix (on a 4.4.0) resolves a kernel null pointer dereference:

--- a/drivers/pci/pcie/aspm.c
+++ b/drivers/pci/pcie/aspm.c
@@ -552,11 +552,12 @@ static struct pcie_link_state *alloc_pcie_link_state(struct pci_dev *pdev)
 void pcie_aspm_init_link_state(struct pci_dev *pdev)
 {
        struct pcie_link_state *link;
-       int blacklist = !!pcie_aspm_sanity_check(pdev);
-
+       int blacklist;
        if (!aspm_support_enabled)
                return;

+       blacklist = !!pcie_aspm_sanity_check(pdev);
+
        if (pdev->link_state)
                return;

Kind of odd that a sanity check is performed if the feature itself is deactivated, the actual null pointer dereference happend in pcie_aspm_sanity_check in this line list_for_each_entry(child, &pdev->subordinate->devices, bus_list) {. Is this a kernel bug?

0

1 Answer 1

0

Is it somehow possible to tell the kernel (e.g. via a parameter) to automatically do this?

Try booting with pci=realloc we had similar issues hotplugging NVMe drives on ancient motherboards until we passed in that parameter.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .