can ping, can't connect by IP - resetting the network fixes it briefly, then it breaks again

Question

I've got one server (CentOS 5), out of a half dozen, that the network flakes out on.

When it's 'dead', I can still ping other servers by IP, but if I try to make any kind of connection also using an IP (ssh, telnet on 53 to test DNS) it just hangs and does nothing. Running /etc/init.d/network restart makes it work again -- for a short amount of time.

I've compared the config against the other severs a dozen times and don't see anything different aside from the server IP.

Prior to today, this network card & cable have been running flawlessly. I changed the way the network is laid out this morning, and this one server is giving me grief while the others all fell right in to place.

The pings are 100% clean, there's no errors in ifconfig. I don't see any errors in any of the main logs, and I'm not really sure how else to try to diagnose the problem.

If anyone has an insight, or guidance on how to troubleshoot further it would be greatly appreciated.

Answer 1

Assuming this is a physical host and not a virtual machine...

Check your switch port packet counts. When a packet is sent to the host the counter should increment. At the same time use tcpdump on the host to see if it's receiving the packet at all.

Look for IP conflicts and arp errors.

Try moving the host to another switch port. Try using a different physical interface on the host. Try a different cable. Check for link negotiation errors (especially immediately when failure occurs).