0

Had to replace an NIC in my server and I cannot get it to go active.

Running RHEL 7 and the device is an AT-2911SX/LC-901. I've done the following:

  • Deleted the rules file in /etc/udev/rules.d
  • Changed the HWADDR in /etc/sysconfig/network-scripts/ifcfg-p1p1
  • Issued reboot command
  • Issued ifconfig -a to see that p1p1 was up with an IP (which it wasnt before)
  • Issued ping 192.168.0.42 which is a box on the same rack connected to the switch; fails
  • Issued ethtool -i p1p1 to see the driver information which was tg3

There are no lights from the rear of the case on the slot where the NIC is. The box at 192.168.0.42 is an identical box but its p1p1 is running bnx2x, not sure if thats the issue.

Any ideas for troubleshooting what else could case it to not ping or connect? The switch it is connected to also shows that it is not TX/RX.

Updates:

Running dmesg | egrep -i -e tg3 -e p1p1 I get the following:

  • IPv6 ADDRCONF(NETDEV_UP) p1p1: link is not ready
  • irq 255 - 275 is MSI/MSI-X
  • eth4 renamed p1p1

Unforunately I cannot directly post output here due to system security. There are also lines for eth0, em1 through em3 and that all have a section like so:

  • WiredSpeed[1], EEE[1], ASF[1]

However the same line for eth4 which becomes p1p1, those values are [0].

3
  • How did it get the IP, and which IP was it? Possibly a 169.. IP? Also do ip route get 192.168.0.42 to see if the routing table is correct, and tcpdump -ni p1p1 to see if you get any traffic at all (at least broadcasts should show up). – dirkt Oct 24 '19 at 21:03
  • @dirkt tcmpdump does nothing on 192.168.0.42 but traffic returns if run on the machine at 192.168.0.42. The IP was set up prior by somebody who did no documentation I am afraid. – datta Oct 28 '19 at 14:09
  • @dirkt The IP is listed in the ifcfg-p1p1 file. Forgot to mention ip route get ... returns the a single line that says cache. A netstat -rn shows in routing table. – datta Oct 28 '19 at 14:35
2

UP is the administrative state indicator, or what you have configured the NIC to do. The corresponding operational state indicator, or what the NIC has actually been able to do, would be RUNNING.

If you see UP but not RUNNING in the ifconfig -a output, it usually means the NIC is not detecting a link from the cable, i.e. the cable is not properly plugged in or is broken.

Since you are talking about a fiberoptic NIC, there is one more thing you should check: if you have connected the NIC's TX to the switch's TX and respectively for RX, then the TX fibre has transmitters in both ends trying to push light into the fibre and no receivers, and the RX fibre is completely dark because it has only receivers connected to it at both ends. The optic cables should always be connected so that the TX at one side ends up to RX at the other side and vice versa.

Since the card is a SX (short-haul) variant, the wavelength of the light used is 850 nm, or a very deep red. With the NIC in the UP state and the cable disconnected, you should see a dim red glow from the TX-side connector.

However, avoid looking directly into the connector with unprotected eyes: that is a bad habit that might cost you your eyesight if you ever accidentally look into a LX (long-haul) fiberoptic. They use 1310 nm laser transmitter, which is infrared (=invisible), and has enough power that it can burn new blind spots into your retinas if it hits them. If you don't have an appropriate tool for detecting the optical signal, give the connector some shade and try to see the reflection of the SX-type light on your hand, a piece of paper, or any non-shiny surface. It's easiest to see at the female connector, but the male end of a connected cable can also produce a tiny faint spot of red light.

If you can't get it working by switching the halves of the pair of fiberoptic cables around, check the optic cables carefully. A kinked fiberoptic cable can easily be fatally damaged. Test with a new cable if you're unsure.


The tg3 driver needs a firmware file - does /lib/firmware/tigon/tg3.bin exist?

If not, get the linux-firmware-20190429-72.gitddde598.el7.noarch.rpm file (or similar) from RHEL installation media or download it directly from RedHat using yum --downloadonly --downloaddir=. linux-firmware in the other RHEL7 system. Copy it to USB stick or some other removable media, move it to the system that has the problem NIC and install with yum localinstall <pathname_of_the_RPM_file>. Then reboot (or disable the network interface, unload & reload the tg3 module and re-enable) to see if it works better now.

If you run dmesg | grep -i -e tg3 -e p1p1, is there any output?

Of course, it is possible that your replacement NIC was bad (Dead On Arrival) and needs to be replaced again.

9
  • NIC shows <UP, BROADCAST, MULTICAST> and not RUNNING. Swapped cables with another running NIC and worked that continued to run fine. Verified TX to RX and still no result. – datta Oct 28 '19 at 14:08
  • @datta I just noticed the NIC needs a firmware file - see updated answer. – telcoM Oct 28 '19 at 14:35
  • The tg3.bin does exist, any special commands I should run with it? – datta Oct 28 '19 at 14:49
  • Not really; the tg3 module should load it automatically after the module is loaded and the NIC is being activated. If there is a problem with the firmware loading, there should be a message in dmesg output saying so. – telcoM Oct 28 '19 at 14:53
  • 1
    dmesg is a command that displays messages from the kernel message buffer. It usually contains a lot of informational/diagnostic messages, but if the kernel or some kernel module has a problem of some sort, the most descriptive error messages from them can usually be found in the dmesg output. – telcoM Oct 28 '19 at 14:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.