For the past week or so, some of my FreeBSD servers intermittently freeze, meaning that their normal network services stop responding. IPMI over the same network continues to work on these frozen boxes. I am successfully rebooting the frozen services via IPMI, and I can see the last console screen via IPMI prior to the freeze. That last console screen does not contain any kind of "panic" output from the kernel.

I don't see any suspicious entries in /var/log/messages or console.log after rebooting.

These servers are all running FreeBSD 8.4-RELEASE-p22 (amd64). There are about 100 servers. They vary in age from about 1 to 5 years.

7 of these machines froze last week, but I do not see any pattern in which machines froze and which stayed running.

How do I diagnose this?

UPD: We found possible solution for strange system freezes. There is some vulnerability in FreeBSD tcp stack. Under it some TCP connections transitioning to the LAST_ACK state can become permanently stuck: https://www.freebsd.org/security/advisories/FreeBSD-SA-15:13.tcp.asc

  • Was there any software upgrade, or change in a workload, that seems correlated in time? When the machine freezes, can you try to ping it? If it respondes, it means at least parts of the kernel are alive; that might suggest running out of resources (memory, in particular). – Edward Tomasz Napierala Jan 27 '15 at 18:20
  • Servers unreacheble over network, including ping. i think out of memory notes on console screen, including console.log. LA under monitoring every minute. We regularly update something at platform, but in this case we have no idea what's maybe wrong. – moronizzz Jan 27 '15 at 19:11
  • Have you considered upgrade? It's always problematic, but might be easier than my second suggestion, which is rebuilding the kernels to include debugger (DDB), and try to enter it after the hang. – Edward Tomasz Napierala Jan 27 '15 at 20:39
  • Please, explain more detail about DDB, it is really possible connecting to it when server completely not responding? – moronizzz Jan 28 '15 at 0:11
  • In some cases - if, for example, the kernel has deadlocked. Few more thoughts: are all the machines in the same location? Did you check for environmental problems, such as overheating? Do you have any network monitoring (IDS) set up? Is there anything unusual in the traffic? – Edward Tomasz Napierala Jan 28 '15 at 11:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.