Intermittent ssh denial

Question

The main server in my network (nothing less...) plays tricks with its ssh connectivity. It is a CentOS 6.4/64 (runs as KVM-based virtualization host with a dozen VMs on it).

It receives its IP via DHCP with fixed assignment. Its /etc/resolv.conf seems just fine, as well as its network co It would normally work, allowing me to ssh into it and being logged in, I'm able to see the internet, ping external servers etc.

But intermittently, it will deny ssh access.

• If pinged from the LAN, it is there; also nmap or opening a socket via bash against port 22, it is there.
• Pinged from outside it won't respond. Currently open ssh sessions stay connected (but now they cannot see the internet nor ping me back).
• The VMs on top of it continue working perfectly, no issues there, all have public IPs and use bridges as per KVM best practices...(afaik).
• Not a single trace in the log about what can be going wrong with it.
• I consider the network to be healthy.
• Modified the sysctl.conf to set net.ipv4.tcp_keepalive_time=120 to no avail.

Then, all of a sudden, it will come back to life and work as if nothing has happened (but it may take up to 10 may be 20 minutes to reappear... - not good for a critical component...)

No iptables in the way (disabled).

Any hints about how to troubleshoot this ?

(This could be a duplicate of SSH getting disrupted intermittently but conditions look somewhat different).