This is rather a basic linux administration question. We have a CentOS linux machine running a production application. There are 10 application specific processes running on that machine.
Once in every 3/4 days, the linux machine freezes and the only way to get it back is to hard reboot it from Amazon AWS console.
We have Amazon Cloudwatch enabled which captures the CPU usage every 5 minutes. We see that the CPU reaches 100% (8 cores) within in 10-15 seconds just before it freezes. And unfortunately we could not figure out anything from the process log files.
How do we really pin point
which process out of those 10 processes is causing the linux server to freeze?
Are there any simple CPU/memory monitoring applications that can record the top cpu/memory hoggers to disk, say every 2 seconds? Appreciate any other ideas to figure out the culprit process.