2

Recently went from Centos 6.4 to Centos 6.5. System is Dell T410 Tower. Whenever I reboot, system runs through post, passes IPv4 Settings, then announces:

PCIe Training Error: Slot 1

system halted

When I ctrl+alt+del from here - I've received same error (pretty sure every time).

Dell ESM Log shows: General failure after video.

When I hit power button, then power back on, system boots as expected.

Slot 1 has a has the nic card the system is using. (After initial Centos 6.4 install, onboard Broadcom nics were giving me speed & latency issues, tried to resolve with drivers, but had no luck, so I added this other nic card (uses Realtek), which worked fine, no issues, until 6.5 update)

I'm still somewhat of a linux noobie, I wouldn't have thought this would kernel related, because I didn't think kernel started loading till next screen. At this point, it seems like it must be the kernel.

I'm running 2.6.32-431.5.1.el6.x86_64 now because I had same issue with 2.6.32-431.11.2.el6.x86_64 PLUS additional CIFS mount issues, which have gone away with downgrade.

uname -r
2.6.32-431.5.1.el6.x86_64

rpm -qa kernel 
kernel-2.6.32-358.23.2.el6.x86_64 
kernel-2.6.32-431.5.1.el6.x86_64 
kernel-2.6.32-358.el6.x86_64 
kernel-2.6.32-431.11.2.el6.x86_64 
kernel-2.6.32-358.14.1.el6.x86_64

I was thinking about trying to go 1 further back to 6.4 kernel to see if reboot yields same result. I'm not sure what additional issues may arise trying to revert just the kernel to previous release (6.5 -> 6.4) & if there are other reasons I shouldn't. I think there may be an option in BIOS (somewhere) to continue after halt messages...but if I do that, and there is an error, seems like chances are nic won't work correctly.

This server acts as a proxy server to the majority of our desktops on the network, so I only have limited timeframes I can power down and test theories. System seems to be running well now, and as long as I don't manually reboot, seems like we'll be ok in current state, but would like to know what the heck is going on.

dmesg snippet:

udev: starting version 147
EDAC MC: Ver: 2.1.0 Feb 12 2014
bnx2: Broadcom NetXtreme II Gigabit Ethernet Driver bnx2 v2.2.3 (June 27, 2012)
  alloc irq_desc for 36 on node -1
  alloc kstat_irqs on node -1
bnx2 0000:01:00.0: PCI INT A -> GSI 36 (level, low) -> IRQ 36
bnx2 0000:01:00.0: setting latency timer to 64
bnx2 0000:01:00.0: firmware: requesting bnx2/bnx2-mips-09-6.2.1b.fw
bnx2 0000:01:00.0: firmware: requesting bnx2/bnx2-rv2p-09-6.0.17.fw
bnx2 0000:01:00.0: eth0: Broadcom NetXtreme II BCM5716 1000Base-T (C0) PCI Express found at mem da000000, IRQ 36, node addr a4:ba:db:12:39:24
  alloc irq_desc for 48 on node -1
  alloc kstat_irqs on node -1
bnx2 0000:01:00.1: PCI INT B -> GSI 48 (level, low) -> IRQ 48
bnx2 0000:01:00.1: setting latency timer to 64
bnx2 0000:01:00.1: firmware: requesting bnx2/bnx2-mips-09-6.2.1b.fw
bnx2 0000:01:00.1: firmware: requesting bnx2/bnx2-rv2p-09-6.0.17.fw
bnx2 0000:01:00.1: eth1: Broadcom NetXtreme II BCM5716 1000Base-T (C0) PCI Express found at mem dc000000, IRQ 48, node addr a4:ba:db:12:39:25
r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded
  alloc irq_desc for 41 on node -1
  alloc kstat_irqs on node -1
r8169 0000:02:00.0: PCI INT A -> GSI 41 (level, low) -> IRQ 41
r8169 0000:02:00.0: setting latency timer to 64
  alloc irq_desc for 62 on node -1
  alloc kstat_irqs on node -1
r8169 0000:02:00.0: irq 62 for MSI/MSI-X
r8169 0000:02:00.0: eth2: RTL8168c/8111c at 0xffffc90000c76000, 00:e0:4d:1a:12:76, XID 1c2000c0 IRQ 62
r8169 0000:02:00.0: eth2: jumbo features [frames: 6128 bytes, tx checksumming: ko]
ACPI Error: No handler for Region [IPMI] (ffff8801ad4ec420) [IPMI] (20090903/evregion-319)
ACPI Error: Region IPMI(7) has no handler (20090903/exfldio-295)
ACPI Error (psparse-0537): Method parse/execution failed [\_SB_.PMI0._GHL] (Node ffff8801ad4eb4c0), AE_NOT_EXIST
ACPI Error (psparse-0537): Method parse/execution failed [\_SB_.PMI0._PMC] (Node ffff8801ad4eb560), AE_NOT_EXIST
ACPI Exception: AE_NOT_EXIST, Evaluating _PMC (20090903/power_meter-759)
  • The message indicates that either the card or slot it is in are broken. I don't see how what kernel you use could affect this. – psusi Apr 14 '14 at 19:50
  • @psusi I tend to agree..just weird timing I guess...any thoughts on why system halt only happens on reboot, but not shutdown -> power button? – webaholik Apr 14 '14 at 20:57
  • 1
    Ahh, in that case, the kernel may be leaving the device in a state it does not reset properly from. – psusi Apr 15 '14 at 0:14
  • @psusi hmm...any thoughts on a work around, like a way to kill power or change state of nic before reboot? Whatever would put it back to working state at boot time. – webaholik Apr 16 '14 at 16:36
1

Either the slot the system in sounds faulty or the NIC itself is having issues coming out of a reboot. Once power has been cut from the board and you initiate the power button reboot the card's state has been cleared, and so is able to boot OK from this state.

Believe it or not I've seen similar issues in the past with hardware that cannot come out of a "warm" state, but can from cold boots.

At any rate I would look to replace the hardware sooner rather then later.

EDIT #1

Looking through the Dell PowerEdge T410 Owners Manual there's this section which shows the various error messages along with suggestions on how to resolve them.

excerpt from page 50

                    ss #1

According to this table it would seem to be agreeing with what I suggested. I would power the system down and attempt to reset the NIC in slot one to see if the error/issue goes away. If it does not then you either have a failing NIC or the motherboard's slot #1 is faulty/failing.

  • This nic is fairly new. It works fine when power off then on. Just not at reboot, psusi mentioned above, "kernel may be leaving the device in a state it does not reset properly from" this seems more likely to me. – webaholik Apr 16 '14 at 16:40
  • @webaholik - see updates. – slm Apr 17 '14 at 3:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.