For the past week or so, some of my FreeBSD servers intermittently freeze, meaning that their normal network services stop responding. IPMI over the same network continues to work on these frozen boxes. I am successfully rebooting the frozen services via IPMI, and I can see the last console screen via IPMI prior to the freeze. That last console screen does not contain any kind of "panic" output from the kernel.
I don't see any suspicious entries in
console.log after rebooting.
These servers are all running FreeBSD 8.4-RELEASE-p22 (amd64). There are about 100 servers. They vary in age from about 1 to 5 years.
7 of these machines froze last week, but I do not see any pattern in which machines froze and which stayed running.
How do I diagnose this?
UPD: We found possible solution for strange system freezes. There is some vulnerability in FreeBSD tcp stack. Under it some TCP connections transitioning to the LAST_ACK state can become permanently stuck: https://www.freebsd.org/security/advisories/FreeBSD-SA-15:13.tcp.asc