I have 15 identical Linux RH 4.7 64-bit severs. They run cluster database (cluster is application level). On occasion (every month or so) a random box (never the same though) freezes.
I can ping the box and ping works. If I try to ssh in the box I get:
ssh_exchange_identification: Connection closed by remote host
SSH is set up properly.
When I go to the server room, and try to login directly to console, I can switch consoles with Alt+Fn, I can enter a username, and characters do show, but after pressing Enter, nothing happens. I waited 8 hours once and it didn't change.
I set up syslog to log everything to a remote host, and there is nothing in those logs. When I reboot the machine, it works without a problem. I have run HW tests - everything is ok, and nothing is in the logs. The machines are also monitored with NAGIOS, and there is no unusual load or activity prior to freeze.
I have run out of ideas; what else can I do or check?